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Preface 

The  purpose  of  this  study  was  to  survey  experts  in  the 
field  of  source  selection  to  determine  if  they  could  agree 
whether  or  not  certain  situations  encountered  during  a 
source  selection  were  pitfalls.  Most  of  the  situations 
tested  were  culled  from  current  source  selection  literature. 
The  participants  also  provided  possible  consequences  for 
each  situation  and  steps  (best  practices)  a  source  selection 
team  could  use  to  avoid  the  situation.  It  is  hoped  that 
this  information  will  help  those  new  to  the  source  selection 
process  avoid  these  pitfalls  or  at  least  anticipate  their 
occurrence . 

I  would  like  to  thank  many  people  for  their  assistance 
during  this  research  effort.  First,  to  my  advisors.  Dr. 
Robert  Pappas  and  Lt  Col  Carl  Templin  for  their  insight  and 
advice  on  developing  this  study  and  carrying  out  the 
research.  Also  the  following  source  selection  officers  at 
the  four  product  centers  were  extremely  helpful  in  securing 
source  selection  experts  willing  to  participate  in  this 
research;  Patrick  Kanoti,  SMC;  Jim  Witham,  ASC;  Don 
Norville,  HSC;  and  Cindy  Burrows,  ESC.  Finally,  thanks  to 
the  survey  participants,  whose  sharing  of  time  and 
experiences  made  this  research  possible. 

Scott  A.  Savoie 
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Abstract 

Source  Selection  is  the  process  used  by  the  Federal 
government  to  select  contractors  for  large,  complex  and 
important  requirements .  Recent  studies  have  indicated  that 
the  inexperience  of  source  selection  personnel  continues  to 
be  a  significant  problem  in  source  selections.  Air  Force 
Federal  Acquisition  Regulation  Appendix  AA  requires  that 
lessons  learned  be  compiled  at  the  conclusion  of  every 
source  selection.  Unfortunately,  there  has  been  little 
success  in  organizing  this  information  int<"  a  coherent 
training  guide.  This  research  attempts  to  fill  this  gap  by 
using  source  selection  experts  to  identify  source  selection 
pitfalls,  their  consequences,  and  the  best  practices  to 
avoid  them.  Fifteen  situations  that  could  be  detrimental  to 
a  source  selection  were  identified  through  a  literature 
review.  Next,  source  selection  experts  were  surveyed  to  see 
if  they  could  agree  on  the  nature  and  effect  these 
situations  had  on  source  selections.  The  experts  evaluated 
each  situation  in  terms  of  its  negative  impact  on  an 
acquisition  program  and  its  frequency  of  occurrence.  The 
experts  were  surveyed  using  the  Delphi  ...athod,  characterized 
by  iterative  survey  rounds  and  feedback  from  previous 
rounds.  The  two  factors  of  impact  and  frequency,  as  well  as 


expert  comments,  were  used  to  identify  which  situations  were 
pitfalls.  Twelve  of  the  fifteen  situations  tested  were 
found  to  be  source  selection  pitfalls.  The  experts  also 
provided  the  possible  consequences  of  and  best  practices  to 
avoid  each  pitfall. 
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AN  INVESTIGATION  INTO  THE  PITFALLS,  CONSEQUENCES  AND  BEST 


PRACTICES  DURING  THE  EVALUATION,  NEGOTIATION  AND  AWARD 
PHASES  OF  A  SOURCE  SELECTION  IN  AFMC  PRODUCT  CENTERS 

I .  RESEARCH  PROBLEM 


Introduction 


In  Fiscal  Year  1993,  Air  Force  Materiel  Command  (AFMC) 
obligated  over  27.9  billion  dollars  in  contracting  actions. 
Of  this  total,  14.6  billion  dollars,  or  52%  was  awarded 
using  competitive  procedures.  Another  30%,  or  $8.5  billion 
were  follow-on  awards  to  previous  competitive  actions  (HILL, 
1994).  Therefore,  over  82%  of  AFMC's  contract  obligations 
have  been  influenced  by  the  competitive  procedures  used  by 
AFMC.  This  thesis  will  look  at  the  competitive  process  used 
for  the  largest  and  most  complex  contract  awards,  source 
selection . 

Source  selection  is  a  term  used  to  describe  a 
formalized  procedure  in  federal  government  contracting.  It 
is  also  used  in  the  commercial  sector  in  a  more  generic 
sense  to  describe  one  of  the  many  function  of  a  purchasing 
department.  A  commercial  purchasing  department  has  many 
roles;  assisting  in  the  development  of  requirements  and 
specifications,  value  analysis,  market  research  and 
administration  of  purchase  orders  and  contracts  are  just  a 
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few  (Dobler,  1990:21,  22).  However,  most  procurement 
experts  cite  the  responsibility  of  selecting  the  right 
source  as  the  most  crucial  role  that  a  purchasing 
organization  fulfills  (Dobler,  1990:30,  209;  Heinritz, 
1986:23;  others) .  In  a  manufacturing  company,  for  example, 
the  purchasing  department  is  the  biggest  user  of  revenue  in 
the  firm,  spending  almost  50%  of  the  revenue  to  buy  material 
necessary  to  fuel  production  operations  (Dobler,  1990:11). 
Thus,  the  success  of  a  company  is  linked  to  the 
effectiveness  of  its  purchasing  department .  It  is  difficult 
for  a  firm  to  operate  efficiently  and  effectively  if  it 
selects  poor  suppliers  to  do  business  with.  A  purchasing 
axiom  reflects  this  view;  "Suppliers  are  an  extension  of  our 
manufacturing  capability"  (Heinritz,  1986:23) . 

Selecting  the  right  source  is  a  critical  role  for  Air 
Force  acquisition  personnel  as  well.  Getting  the  best  value 
for  the  dollar  continues  to  be  an  important  goal  for  the  Air 
Force,  especially  with  the  continuing  decline  in  budgets . 
And,  getting  the  best  value  starts  with  selecting  the  right 
source . 

The  purpose  of  a  source  selection  is  to  choose  the 
contractor  that  will  best  fulfill  the  government's 
requirement.  The  result  of  the  source  selection  process  is 
a  contract  award,  that  can  be  worth  millions  (or  even 
billions)  of  dollars.  This  researcher  thinks  that  the 
purchasing  axiom  previously  stated  is  clearly  applicable  to 

Department  of  Defense  contracting  with  minor  modifications; 
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"Defense  contractors  are  an  extension  of  our  warfighting 
capability . " 

Consider  the  acquisition  of  the  F-22,  Advanced 
Tactical  Fighter  (ATF) .  On  2  August,  1991,  the  Air  Force 
awarded  a  contract  to  the  contractor  team  of  Lockheed/Boeing 
for  the  Engineering  Manufacturing  Development  (EMD)  phase  of 
the  ATF  program.  This  contract  to  develop  the  F-22  air 
vehicle  was  worth  over  $9.5  billion  (Hatfield,  1994).  This 
contract  committed  the  Air  Force  to  the  Lockheed/Boeing  team 
through  the  end  of  the  EMD  phase,  until  approximately  2003 
(Raggio,  1994)  .  This  commitment  will  undoubtedly  continue 
under  a  production  contract,  worth  billions  more,  should  the 
ATF  program  go  into  production.  The  process  used  to  choose 
the  Lockheed/Boeing  team  over  it ' s  competitor,  the 
Northrop/McDonnell  Douglas  team  was  source  selection. 

It  is  clear  that  selecting  the  right  source  is 
critical  to  the  Air  Force,  just  as  it  is  to  the  commercial 
sector.  It  is  therefore  important  that  Air  Force 
acquisition  professionals  thoroughly  understand  the  source 
selection  process. 

Bflc.kg:rQund 

In  the  Air  Force,  source  selection  procedures  are 
required  on  the  following  competitively  negotiated 
procurements : 

(1)  Major  Defense  Acquisition  Program  as 
defined  in  DoDD  5000 . 1 . 
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(2)  Each  new  development  or  production 
program  designated  by  the  Assistant  Secretary 
of  the  Air  Force  for  Acquisition. 

(3)  Communication  and  Computer  Programs 
estimated  at  $200  million  or  more,  or, 

(4)  Modification,  Maintenance,  or 
Services  or  projects  estimated  to  exceed  $750 
million.  (AFFARS,  Appendix  AA,  1993:3) 

The  objective  of  the  source  selection  process  is  to 
"select  the  source  whose  proposal  has  the  highest  degree  of 
credibility  and  whose  performance  can  be  expected  to  best 
meet  the  government ' s  requirement  at  an  affordable  cost" 
(AFFARS,  Appendix  AA,  1993:3).  Although  cost  will  always  be 
an  evaluation  factor  for  any  type  of  government  purchase,  a 
source  selection  procurement  will  invariably  include  an 
evaluation  of  each  contractor's  unique  approach  to  fulfill 
the  government ' s  requirement .  Depending  on  the  government ' s 
need,  a  contractor's  proposal  may  include  a  particular 
technical  approach,  manufacturing  plan,  or  use  of  certain 
materials  and  equipment.  Many  different  functional  experts 
converge  to  form  the  Air  Force  evaluation  teams .  Teams  may 
include  contract  specialists,  engineers,  cost  analysts, 
logisticians;  in  short,  any  discipline  which  will  be 
evaluated  in  the  source  selection  will  be  represented. 

Erablem  Statement 

As  important  as  a  source  selection  is  to  an 
acquisition  program.  Air  Force  officials  are  not  always 
adequately  trained  prior  to  participating  in  a  source 
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selection.  Two  AFIT  theses  have  cited  inexperienced  source 
selection  personnel  as  a  major  problem  during  the  source 
selection  process  (Gray  and  Hugo,  1985:6-1;  Babcock,  1986:5- 
6) .  More  recently,  in  her  thesis  An  analysis  of  the  Source 
Selection  Process  at  Aeronautical  Systems  Division,  Elaine 
C.  Rourke  cited  a  continued  lack  of  experience  of  source 
selection  personnel  as  a  major  problem  in  the  source 
selection  process  (Rourke,  1989:106).  In  fact,  it  was  the 
most  frequently  cited  problem  in  her  study.  One  of  Ms. 
Rourke 's  recommendations  was  that  a  core  source  selection 
team  be  established  to  conduct  major  source  selections  for 
Aeronautical  Systems  Division  (now  Aeronautical  Systems 
Center  or  ASC) .  This  team  would  become  the  source  selection 
experts  for  the  center  and  would  ensure  that  no  source 
selection  suffered  from  lack  of  experience. 

One  improvement  implemented  by  ASC  was  the 
establishment  of  the  source  selection  support  office 
(ASC/CYX) .  This  office  provides  facilities  and  training  to 
center  personnel  before  they  participate  in  a  source 
selection  and  also  provides  them  with  facilities  in  which  to 
conduct  the  source  selection.  Support  offices  also  exist  at 
AFMC's  Space  and  Missile  Center  and  Electronic  Systems 
Center.  However,  a  core  source  selection  team  does  not 
exist  at  any  of  the  product  centers  and,  with  the  current 
downsizing  trend,  it  is  highly  unlikely  that  they  ever  will. 

This  research  is  aimed  at  improving  the  knowledge 

base  and  experience  of  source  selection  teams  in  another 
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way;  by  tapping  into  the  views  and  opinions  of  source 
selection  experts  who  have  participated  in  numerous  source 
selections .  By  documenting  their  experiences  and  sharing 
them  with  other  acquisition  professionals,  the  current 
problem  of  inexperienced  source  selection  personnel  may  be 
reduced.  These  experts  may  be  able  to  identify  pitfalls 
which  any  source  selection  could  encounter,  the  consequences 
to  the  pitfalls  and  the  best  practices  to  avoid  them.  Then, 
less  experienced  source  selection  teams  and  individuals 
could  use  this  information  to  successfully  navigate  their 
way  through  the  complex  source  selection  process . 

A  similar  product  has  already  been  developed  for  the 
process  of  transitioning  from  development  to  production. 
Informally  called  the  Willoughby  Templates,  this  guide 
identifies  traps,  or  management  practices  that  result  in 
high  risk  to  program  success .  The  guide  further  identifies 
the  consequences  of  these  traps  and  ways  (best  practices)  to 
avoid  them  (Best  Practices,  1986) . 

The  intent  of  this  research  i3  to  determine  if  the 
source  selection  process  lends  itself  to  this  kind  of  study; 
or  whether  each  source  selection  is  so  unique  that  it 
carries  with  it  unique  pitfalls  that  are  not  a  threat  to 
other  source  selections .  Consequently,  the  purpose  of  this 
research  is  to  determine  if  there  are  common  pitfalls  any 
source  selection  may  encounter,  the  consequences  of  these 
pitfalls,  and  best  practices  to  avoid  these  pitfalls. 
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This  research  seeks  to  determine  if  there  are 


pitfalls  that  are  common  to  AFMC  product  centers .  To  do 
this  the  study  will  investigate  the  following  research 
objectives : 

1.  Examine  the  extent  to  which  source  selection  experts  can 
agree  on  situations  that  could  be  detrimental  to  a  source 
selection . 

2.  Examine  the  impact  these  situations  have  on  the 
acquisition  program  and  their  frequency  of  occurrence. 

3.  Examine  the  extent  to  which  these  pitfalls  are  unique  to 
particular  product  centers  or  common  to  all  Air  Force 
Material  Command  (AFMC)  product  centers . 

4.  Determine  what  the  consequences . are  to  the  acquisition 
process  if  one  of  these  pitfalls  is  encountered. 

5.  Determine  the  extent  to  which  Air  Force  source  selection 
personnel  can  take  certain  steps  (best  practices)  to  avoid 
these  pitfalls  before  they  occur. 

The  first  research  objective  will  help  determine  the 

degree  to  which  source  selection  experts  can  agree  on 

whether  or  not  a  given  situation  is  a  pitfall.  Research 

Objective  #2  will  attempt  to  measure  this  agreement  in  two 

ways;  impact  and  frequency.  These  first  two  research 

objectives  will  provide  the  answer  to  whether  or  not  a 

situation  is  a  pitfall.  Research  Objective  #3  will  look  to 

see  if  there  are  any  differences  between  product  centers  on 

the  impact  and  frequency  of  these  situations .  Finally, 
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Research  Objectives  4  and  5  will  describe  the  consequences 
and  best  practices  for  those  situations  that  are  found  to  be 
pitfalls . 

Scope  and  Limitation  of  Research 

The  scope  of  this  investigation  will  be  the  four  Air 
Force  Materiel  Command  (AFMC)  product  centers,  Aeronautical 
Systems  Center  (ASC) ,  Electronic  Systems  Center  (ESC) ,  Space 
and  Missile  Center  (SMC)  and  Human  Systems  Center  (HSC) . 

This  research  effort  will  limit  its  investigation  to 
the  pitfalls  encountered  during  the  evaluation,  negotiation 
and  award  phases  of  the  process .  While  the  best  practices 
to  avoid  these  pitfalls  may  be  taken  before  this  time,  the 
actual  pitfalls  will  occur  during  one  of  these  phases . 

Finally,  only  pitfalls  attributable  to  the  structure, 
organization  and  process  of  a  source  selection  and  the  Air 
Force  acquisition  process  will  be  investigated.  There  are 
potential  psychological  pitfalls,  such  as  personality  type, 
which  can  affect  any  type  of  negotiation,  from  arms  control 
treaties  to  a  car  dealer  negotiation.  It  is  not  the  intent 
of  this  research  to  investigate  these  types  of  pitfalls. 
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This  section  describes  the  content  of  the  subsequent 
chapters  in  this  thesis.  Chapter  2,  Literature  Review, 
discusses  the  review  of  the  source  selection  literature  that 
was  accomplished  for  this  study.  It  describes  the  source 
selection  process  from  receipt  of  proposals  to  contract 
award  and  discusses  the  potential  pitfalls  that  may  be 
encountered  along  the  way.  Chapter  2  concludes  with  a  list 
of  potential  pitfalls  that  was  found. 

Chapter  3,  Methodology,  describes  the  method  by  which 
the  researcher  collected  data  on  the  potential  pitfalls, 
consequences  and  best  practices .  It  describes  the  Delphi 
technique  used  in  this  study  and  discusses  why  it  was  an 
appropriate  method.  It  also  discusses  the  construction  and 
administration  of  the  surveys  used  for  this  study. 

Chapter  4,  Analysis,  provides  an  analysis  of  the  data 
that  was  collected.  It  contains  a  detailed  discussion  on 
each  situation  that  was  tested.  Each  discussion  addresses 
the  research  objectives  as  they  relate  to  that  particular 
situation . 

Finally,  Chapter  5,  Conclusion,  summarizes  the  findings 
of  the  research  as  well  as  its  limitations  and 
recommendations  for  further  research. 


This  chapter  describes  the  literature  review  that  was 
accomplished  for  this  research  effort.  First,  it  will  cite 
sources  that  describe  the  overall  source  selection  process. 
Next,  it  will  look  at  the  process  in  detail  starting  when 
the  government  receives  proposals  until  contract  award. 
During  this  detailed  description,  potential  pitfalls  will 
also  be  described.  Finally,  the  chapter  will  recap  the 
source  selection  situations  found  in  the  current  literature 
that  could  be  potential  source  selection  pitfalls . 


There  have  been  several  theses  recently  that  have  dealt 

with  source  selection.  Two  of  them  contain  descriptions  of 

the  overall  source  selection  process.  (Babcock,  1986:  2-1  to 

2-8;  Rourke,  1989:5-23).  Another  authoritative  source  on 

the  source  selection  process  is  an  article  written  by  Dr. 

Curtis  R.  Cook  and  Vernon  J.  Edwards  (Cook  and  Edwards, 

1993 :Chp  42).  Readers  who  are  unfamiliar  with  the  source 

selection  process  should  reference  one  of  these  sources. 

Figure  1  represents  the  conventional  source  selection 

process.  The  shaded  portion  of  Figure  2.1  represents  the 

steps  in  the  source  selection  process  that  will  be 

investigated  in  this  study.  It  start  with  the  receipt  of 

contractor  proposals  and  ends  with  a  contract  award. 
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Figure  2.1  -  Flow 
(Training  Slide, 


chart  of  Source  Selection  Process 
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Once  proposals  are  received,  the  government  begins  it's 
evaluation.  The  purpose  of  this  evaluation  is  to  determine 
to  what  degree  each  proposal  meets  the  requirements  in  the 
Request  for  Proposal  (RFP) .  The  evaluation  assesses  the 
proposal's  strengths,  weaknesses  and  the  risks  associated 
with  accepting  it  (AFFARS,  Appendix  AA,  1993:5).  In  effect, 
the  evaluation  serves  as  a  foundation  to  assess  the 
probability  of  a  contractor  to  successfully  complete  the 
contract  if  selected  for  award  (Cook  and  Edwards, 
1993:42.16).  This  initial  evaluation  is  also  used  by  the 
contracting  officer  to  determine  whether  or  not  discussions 
are  necessary  before  awarding  a  contract  (Source  Selection 
Workshop,  1989:B-55) .  This  is  done  by  identifying 
deficiencies  and  issues  that  require  clarification  in  each 
proposal.  Deficiency  Reports  (DRs)  are  written  on  issues 
that  make  a  proposal  unacceptable  and  Clarification  Requests 
(CRs)  are  written  for  issues  that  need  further 
clarification.  If  the  proposals  received  contain  enough 
issues  that  are  unresolved  or  deficient,  the  contracting 
officer  will  probably  make  the  decision  to  hold  discussions. 

There  have  been  several  problems  identified  during  this 
initial  evaluation.  The  Federal  Acquisition  Regulation 
(FAR)  requires  that  each  proposal  be  evaluated  on  factors 
specified  in  the  RFP  (FAR,  1990:15-13).  One  danger  that  may 
occur  is  if  an  evaluator  evaluates  a  proposal  based  on  a 


comparison  with  other  proposals,  rather  than  on  the 
standards  specified  in  the  RFP .  (Crumbie,  1990:21). 

Another  potential  pitfall  concerns  ratings.  During 
proposal  evaluations,  the  evaluators  also  rate  each 
proposal.  The  typical  method  used  by  the  Air  Force  to 
evaluate  Proposals  is  to  use  a  color  code  system  as  shown  in 
Table  2 . 


Table  2.1  -  Air  Force  Color  Rating  System 

(AFFARS,  Appendix  AA,  1993:AA-15) 


COLOR 

RATING 

DEFINITION 

Blue 

Exceptional 

Exceeds  specified  performance  or 
capability  in  a  beneficial  way  to  the 
Air  Force;  and  has  high  probability  of 
satisfying  the  requirement;  and  has  no 
significant  weakness. 

Greer. 

Acceptable 

Meets  evaluation  standards;  and  had 
good  probability  of  -satisfying  the 
requirement;  and  any  weaknesses  can  be 
readily  corrected. 

Yellow 

Marginal 

Fails  to  meet  evaluation  standards; 
and  has  low  probability  of  satisfying 
the  requirement;  and  has  significant 
deficiencies  but  correctable 

Red 

Unacceptable 

Fails  to  meet  a  minimum  requirement; 
and  deficiency  requires  a  major 
revision  to  the  proposal  to  make  it 
correct . 

The  concern  here  is  that  proposals  with  similar  strengths 
and  weaknesses  receive  different  ratings.  (ASC,  1993:21). 
This  could  be  because  different  evaluators  rate  the  same 
factor  in  different  proposals  or  because  one  evaluator 
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begins  to  rate  proposals  harder  or  more  leniently  as  the 
evaluations  progress. 

Problems  may  also  impact  the  source  selection 
process  during  the  CR/DR  generation  processes.  The  first  is 
the  lengthy  review  and  modification  of  a  CR/DR  before  it  is 
approved.  This  may  be  due  to  the  ignorance  of  the  evaluator 
on  what  is  required  in  a  CR/DR,  resulting  in  the  need  for 
correction.  These  corrections  may  act  to  change  the 
original  meaning  of  the  CR/DR  and  thus  not  alert  the  offeror 
to  the  issue  that  the  evaluator  intended  to  address  (ASC, 
1993:  47).  Another  problem  during  CR/DR  generation  may 
occur  when  a  high  number  of  CR/DRs  are  written  with  many 
redundant  write  ups  (ASC,  1993:24) .  This  again  may  create 
delays  in  the  source  selection  process  as  unnecessary  write 
ups  are  generated  and  responses  to  them  are  developed, 
received  and  reevaluated. 

Finally,  during  initial  evaluation,  many  evaluators  sea 
only  a  portion  of  the  offeror's  proposal.  .The  evaluators 
lack  the  global  view  of  each  offer  and  thus  are  not  able  to 
reconcile  their  evaluations  with  the  big  picture  (ASC, 
1993:21).  This  situation  can  cause  the  evaluation  process 
to  miss  either  conflicting  or  clarifying  information.  If 
this  happens,  then  a  CR/DR  may  not  be  written  to  address 
proposal  information  that  conflicts  or,  a  CR/DR  may  be 
written  on  an  issue  that  the  proposal  has  already  addressed. 
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Performance  Risk  Assessment  Pitfall 

During  the  initial  evaluations,  not  only  are  technical 
evaluations  occurring,  but  an  assessment  of  each  offeror' s 
past  performance  is  also  made.  This  past  performance 
assessment  is  used  to  gauge  the  risk  associated  with 
selecting  a  contractor  using  past  performance  as  a 
criterion.  According  to  AFMC  FAR  Supplement  to  Appendix  AA, 
"Performance  risk  is  a  confidence  measure  that  assesses  the 
offeror' s  present  and  past  work  record  in  order  to  determine 
the  offeror' s  ability  to  perform  the  proposed  effort" 
(AFARSUP  AA:  AA-1) . 

The  source  selection  team  that  conducts  the  performance 
risk  analysis  is  called  the  Performance  Risk  Analysis  Group 
(PRAG) .  They  collect  past  performance  information  in  a 
variety  of  ways:  from  the  Contractor  Performance  Assessment 
Report  (CPARS)  system,  (an  AFMC  data  base)  or,  through 
questionnaires,  interviews  or  another  performance  reporting 
systems  (AFARSUP  AA:  AA-5) . 

The  PRAG  may  rely  on  the  offerors  to  supply  information 
on  past  performance  relevant  to  the  source  selection  (ASC, 
1993:29).  This  may  present  a  potential  problem  to  the  PRAG. 
An  offeror  may  withhold  information  on  past  performance  it 
considers  relevant  to  the  source  selection  but  unfavorable. 
If  this  information  isn't  picked  up  by  the  PRAG  through 
other  sources  then  the  performance  risk  assessment  may  be 
lower  for  this  offeror  than  is  warranted. 
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After  the  initial  proposal  evaluation  the  contracting 
officer  decides  whether  or  not  discussions  are  necessary 
(Source  Selection  Workshop,  1989:B-55) .  This  is  an 
important  step  because  a  significant  amount  of  time  can  be 
saved  if  discussions  are  not  held.  This  was  recognized  back 
in  1987  by  then  Air  Force  Systems  Command  when  it  amended 
Air  Force  Systems  Command  Regulation  550-23  to  encourage 
award  without  discussions  whenever  possible  (Gotcher  and 
Templin,  1993:4).  The  award  without  discussion  policy  is 
still  evident  today  (ASC,  1993:39).  This  emphasis,  however, 
may  become  a  problem  if  it  pressures  a  contracting  officer 
to  avoid  discussions  when  he/she  finds  that  discussions  are 
probably  warranted  (Cook  and  Edwards,  1993:42.20).  This 
policy  may  lead  the  decision  to  fix  the  issue  after  contract 
award  rather  than  conducting  discussions  with  all 
competitive  offerors.  This  would  put  the  Air  Force  at  a 
disadvantage  in  subsequent  negotiations  to  fix  the  contract 
since  the  contractor  is  no  longer  in  the  competitive,  source 
selection  environment .  A  greater  risk  may  be  the 
sustainment  of  a  protest  if  another  offeror  can  show  that 
the  Air  Force  should  have  conducted  discussions. 


If  the  decision  is  made  to  conduct  discussions,  a 

determination  must  be  made  by  the  contracting  officer  on 

which  offerors  to  include  in  the  competitive  range.  Only 
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those  offerors  in  the  competitive  range  will  be  able  to 
participate  in  the  discussions  (FAR,  1990:15-14).  The  other 
offerors  will  be  eliminated  from  further  consideration.  The 


FAR  states  that  if  the  contracting  officer  "determines  that 
a  proposal  no  longer  has  a  reasonable  chance  of  being 
selected"  it  may  be  excluded  from  the  competitive  range 
(FAR,  1990:  15-14). 

The  number  of  offerors  kept  in  the  competitive  range  is 
important.  The  government  will  have  to  conduct  negotiations 
with  all  offerors  in  the  competitive  range  and  ultimately 
review  and  evaluate  the  changes  that  each  of  them  makes  to 
their  proposal.  A  possible  problem  under  these 
circumstances  could  occur  if  the  determination  of  the 
competitive  range  is  made,  in  part,  by  a  motivation  to  keep 
the  number  of  remaining  proposals  low  rather  than  on  the 
"reasonable  chance  of  being  selected  for  award"  criterion. 
(Gilbreth  and  Horst,  1994:B-39). 

Discus  sian, -Ei.  trails. 

After  the  competitive  range  has  been  established,  the 
government  will  enter  into  discussions  with  each  remaining 
offeror.  The  purpose  of  discussions  is  to  maximize 
competition  by  allowing  offerors  to  raise  their  proposal  to 
an  acceptable  level.  To  do  this  the  government  must  be 
specific  about  advising  each  offeror  about  of  the 
deficiencies  in  his/her  proposal  (Source  Selection  Workshop, 
1989:B-77) .  The  FAR  allows  these  discussions  with  offeror 


to  be  written  and/or  oral  (FAR,  1990:  15-15)  .  Written 
discussions  usually  take  the  form  of  DRs  or  CRs . 

There  are  several  potential  pitfalls  that  may  occur 
during  discussions.  First,  the  government  is  required  to 
provide  meaningful  discussions  with  each  contractor.  If  the 
government  fails  to  point  out  a  deficiency  to  an  offeror 
during  discussions,  then  the  discussions  might  not  be  found 
meaningful  (Source  Selection  Workshop,  1989:B-81) .  The 
second  possible  discussion  problem  occurs  when  both  written 
and  oral  discussions  are  conducted.  In  some  source 
selections,  the  government  will  provide  each  offeror  with 
the  DRs  and  CRs  and  later  discuss  these  issues  face  to  face. 
A  member  of  the  government  negotiation  team  may  discuss 
aspects  of  the  proposal  other  than  those  addressed  in  the 
CRs  and  DRs .  If  this  happens  there  is  a  risk  he/she  may 
undermine  the  government's  position  (Crumbie,  1990:26,  27). 
Further,  straying  too  far  from  the  CRs  and  DRs  may  lead  into 
such  prohibited  acts  as  disclosing  aspects  of  competing 
offers,  called  technical  transfusion  (Source  Selection 
Workshop,  1989:B-80) .  Even  if  the  focus  is  on  the  written 
communication,  forbidden  actions  such  as  technical 
transfusion,  technical  leveling  and  auctioning  could  occur 
(Crumbie,  1990:27) .  Technical  leveling  is  when  the 
government  helps . . . 

"...an  offeror  to  bring  it's  proposal  up  to  the  level  of 
other  proposals  through  successive  rounds  of  discussion,  by 
pointing  out  weaknesses  resulting  from  the  offerors  lack  of 
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diligence  [or]  competence"  (Source  Selection  Workshop, 

1989 :B-79) . 

Auctioning  occurs  when  the  government  indicates  a  price 
level  an  offeror  must  reach  to  get  further  consideration  or 
when  the  government  reveals  the  price  of  other,  competing 
offers  (Source  Selection  Workshop,  1989:B-88) . 

BAFO  Pitfall 

After  discussion  are  concluded,  the  government  will  ask 
for  all  offerors  to  submit  their  Best  and  Final  Offers 
(BAFO) .  A  problem  that  may  arise  at  this  point  is  if  all 
the  BAFOs  exceed  the  funds  available  (ASC,  1993:30).  If 
this  occurs  then  additional  funds  would  have  to  be  secured 
or  negotiations  would  have  to  be  reopened  and  the  scope  of 
the  requirement  reduced. 

Mlsce.13.angou.s_  Pitfall 

Finally,  there  may  be  a  potential  problem  that  is  not 
part  of  the  source  selection  process  itself,  but  may  have  an 
impact  on  it.  The  problem  is  when  new  government  personnel 
are  introduced  into  the  evaluation/negotiation  phase  of  the 
source  selection  and  were  not  involved  during  the 
development  of  the  RFP  and  award  criteria  (ASC,  1993:39, 

44)  . 
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The  following  is  a  recap  of  the  possible  source 
selection  pitfalls  that  were  found  during  the  review  of  the 
source  selection  literature.  They  will  be  the  focus  of 
further  study: 

1.  Evaluating  a  proposal  based  on  another  proposal,  and  not 
the  standards  in  the  RFP  (Crumbie,  1990:21). 

2.  Similar  strengths  and  weaknesses  between  proposals 'get 
rated  differently  (ASC,  1993:21). 

3.  Numerous  rewrites  of  a  CR/DR  before  it  is  approved, 
sometimes  changing  its  original  meaning  (ASC,  1993:47). 

4.  High  number  of  redundant  deficiency  reports, 
clarification  requests,  and  modification  requests  (CR/DR) 
are  generated  (ASC,  1993:24). 

5 .  Evaluators  lack  global  view  of  proposal  and  do  not  pick 
up  conflicting  or  clarifying  information  (ASC,  1993:21). 

6.  Past  performance  problems  which,  though  relevant  to  the 
source  selection,  are  unknown  to  or  hidden  from  the  PRAG 
(ASC,  1993:29). 

7.  Sticking  by  an  'award  without  discussions'  policy  when 
it  is  found  that  discussions  are  probably  warranted  (Cook 
and  Edwards,  1993:42.20) . 

8 .  The  motivation  during  the  competitive  range 
determination  is  to  reduce  the  number  of  offerors,  not  to 
retain  offerors  whose  proposals  have  a  reasonable  chance  of 
being  selected  for  award  (Gilbreth  and  Horst,  1994:B-39). 
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9.  Not  all  proposal  deficiencies  are  addressed  during 
discussions  with  offerors  (Source  Selection  Workshop, 

1989 : B-81) . 

10.  Addressing  issues  during  face  to  face  discussions  which 
were  not  addressed  with  a  CR/DR  (Crumbie,  1990:26,  27). 

11.  Technical  leveling,  transfusion  or  auctioning  during 
negotiations  (Crumbie,  1990:27). 

12.  BAFO  prices  exceed  funds  available  (ASC,’  1993:30)  . 

13.  Bringing  people  into  the  evaluation/negotiation  phase 
of  the  source  selection  who  were  not  involved  in  the 
development  of  the  RFP  and  award  criteria  (ASC,  1993:39, 

44)  . 

Summary 

This  chapter  reviewed  the  literature  search  that  was 
accomplished  for  this  study.  First,  it  cited  sources  that 
contain  descriptions  of  the  overall  source  selection 
process.  Next,  it  looked  at  the  process  in  detail  from 
proposal  receipt  until  contract  award,  identifying  possible 
pitfalls  along  the  way.  Finally,  it  recapped  the  potential 
pitfalls  identified  in  the  literature  review  that  will  be 
studied  further.  The  next  chapter  will  describe  the 
methodology  used  to  investigate  these  potential  pitfalls . 
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III. 


IntjQdustiQii 

This  chapter  describes  the  methodology  used  to  conduct 
this  study.  As  stated  previously,  the  intent  of  the  study 
was  first  to  determine  if  there  were  common  pitfalls  which 
any  source  selection  could  encounter.  If  common  pitfalls 
were  present,  the  research  investigated  the  possible 
consequences  to  each  pitfall  and  the  best  practices  to  avoid 
it.  AFFARS  Appendix  AA  requires  that  a  source  selection 
lessons  learned  be  compiled  at  the  conclusion  of  each  source 
selection  (AFFARS,  Appendix  AA,  1993:22).  Unfortunately, 
there  has  been  little  success  in  organizing  this 
information  into  a  coherent  training  guide.  The  existing 
literature  therefore  doesn't  describe  explicitly  the 
pitfalls  one  can  expect  in  a  source  selection.  This  fact 
drove  the  need  for  a  two-stage  research  design  study  (Emory 
and  Cooper,  1991:147).  The  first  stage  was  exploratory  with 
the  objective  of  identifying  the  potential  pitfalls.  The 
second  stage  was  a  formal  survey  to  test  the  existence  of 
these  suspected  pitfalls,  their  consequences,  and  the  best 
practices  to  avoid  them. 

Esaearch  Design 

The  type  of  data  collected  was  ex  post  facto.  In  both 
stages  the  researcher  relied  on  the  past  experiences  of  the 
participants.  The  type  of  study  was  descriptive,  attempting 
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to  describe  through  past  experience  the  common  pitfalls 
associated  with  source  selections.  Also,  there  was  no 
ability  to  manipulate  the  variables  so  a  causal  study  was 
not  possible  to  conduct.  The  time  dimension  of  this  study 
was  cross-sectional  in  nature.  However,  participants 
responded  to  both  stages  of  the  study  based  on  the  source 
selection  experience  they  gained  throughout  their  career. 
Although  this  research  was  not  a  case  study,  it  was  more  in- 
depth  than  a  standard  survey.  This  was  die  :o  two  factors: 
iterative  survey  rounds  and  feedback  from  previous  rounds . 
This  will  be  discussed  in  more  depth  under  the  Delphi 
technique  discussion.  The  survey  employed  was  administered 
using  the  Total  Design  Method  (TDM)  (Dillman,  1978:21).  The 
environment  in  which  the  research  was  conducted  was  a  field 
study,  since  all  participants  completed  the  survey  at  their 
place  of  work. 

Stage  X  -  Explora-tacy  JZhaae 

This  phase  was  conducted  through  an  informal  experience 
survey.  This  is  appropriate  if  published  work  on  the 
research  topic  is  scant  or  difficult  to  find  (Emory  and 
Cooper,  1991:146).  This  survey  was  very  flexible  and 
allowed  participants  latitude  to  fully  cover  the  research 
topic.  First  they  were  given  a  list  of  potential  pitfalls 
identified  during  the  literature  review  and  asked  if  they 
agreed  whether  or  not  each  was  a  probable  pitfall.  Next, 


3-2 


each  participant  was  asked  to  add  additional  source 
selection  situations  which  he/she  felt  was  a  pitfall  but 
wasn't  identified  during  the  literature  review.  This  served 
to  strengthen  the  content  validity  of  the  resulting  Delphi 
survey . 

As  a  result  of  the  experience  survey  two  more  potential 
pitfalls  were  added  and  a  pitfall  already  identified  in 
Chapter  2  was  modified  to  have  broader  coverage.  The 
additions  were: 

1 .  The  price  or  costs  analysis  technique  developed  for  the 
cost/price  evaluation  renders  all  or  most  offers  unrealistic 
and  unreasonable  when  applied. 

2 .  Trying  to  accomplish  the  source  selection  within  an 
arbitrarily  determined  schedule. 

Also,  the  following  potential  pitfall  identified  in  Chapter 

II,  "Evaluating  a  proposal  based  on  another  proposal,  not 
the  standards  in  the  Request  for  Proposal  RFF . "  Was  changed 
to  read,  "A  technical  team  member  evaluates  a  proposal  based 
on  standards  not  in  the  Request  for  Proposal  (RFP) ." 

The  change  on  this  potential  pitfall  made  the  situation 
broader.  Now  the  pitfall  can  occur  whenever  an  evaluator 
uses  standards  other  than  those  described  in  the  RFP  to  do 
his  evaluation.  These  other  standards  may  or  may  not  be 
based  on  another  proposal . 


3-3 


"The  Delphi  method  is  a  name  that  has  been  applied  to  a 
technique  used  for  the  elicitation  of  opinions  with  the 
object  of  obtaining  a  group  response  from  a  panel  of 
experts"  (Brown,  1968:3) .  One  way  to  arrive  at  a  group 
opinion  is  through  face  to  face  discussions.  However, 
research  into  the  accuracy  of  group  opinion  points  out 
several  weaknesses  of  face  to  face  discussions: 

1.  The  influence  of  a  dominant  individual. 

2.  Noise.  Interaction  that  is  aimed  at 
maintaining  the  group  rather  than  effectively 
solving  the  problem. 

3.  Group  pressure  for  conformity. 

(Dalkey : 7 ) 

Research  has  found  that  face  to  face  discussions  are 
often  less  accurate  than  an  average  group  opinion  without 
discussion  (Dalkey: 7).  Further,  when  an  additional  round  of 
surveying  is  added  with  feedback  from  the  previous  round, 
accuracy  is  again  increased  (18:12).  These  are  both 
characteristics  of  the  Delphi  technique.  The  Delphi 
technique  is  best  suited  for  problems  that  are  ill  defined 
or  complex  (Emory  and  Cooper: 76) .  Because  so  little 
research  had  been  done  on  investigating  and  organizing 
potential  source  selection  pitfalls,  the  Delphi  technique 
was  well  suited  for  this  study. 


Information  gathered  using  Delphi  is  done  through  a 
series  of  surveys  or  questionnaires.  Respondents  are  asked 
to  give  reasons  for  their  stated  opinions  and  these  reasons 
are  fed  to  all  participants  in  subsequent  rounds.  This  is 
intended  to  let  respondents  reflect  on  their  previous 
response  based  on  the  informed  judgment  of  others .  This 
allowed  for  a  controlled  debate  among  respondents  without 
the  defects  of  face  to  face  discussion.  (Brown: 3) 

Stags  ..I L,-— Tflxmal .  .Survey 

The  second  stage  of  the  study  consisted  of  developing 
and  pre-testing  the  survey  instrument,  and  conducting  the 
formal  Delphi  survey . 

Survey:  Construction  ^.Saund  Qne 

The  pitfalls  identified  in  Stage  I,  plus  the  research 
objectives,  were  used  to  develop  the  survey.  Appendix  A 
contains  this  round  one  survey.  To  determine  whether  a 
source  selection  situation  was  in  fact  a  pitfall,  two  closed 
ended  questions,  called  factors,  were  asked  of  each 
potential  pitfall: 

1.  If  this  situation  occurs,  what  is  the 
impact  on  the  acquisition  program? 

2.  What  is  the  likelihood  that  this  situation 
could  occur? 

The  participants  responded  to  these  factors  using  the 
following  Likert  scales: 
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Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 

Initially,  the  researcher  was  going  to  use  a  Likert  type 
scale  for  the  first  factor  similar  to  the  one  suggested  by 
Kerlinger  'Kerlinger,  1986:460).  This  was  a  five  point 
scale  from  Strongly  Agree  tc  Strongly  Disagree.  This, 
however,  would  not  capture  the  information  that  the 
researcher  was  seeking.  A  survey  participant  may  be  able  to 
Strongly  Agree  that  two  different  situations  are  both 
pitfalls  in  the  source  selection  process.  But,  he/she  may 
be  agreeing  that  one  of  the  situations  would  probably  have  a 
minor  impact  on  the  acquisition  while  the  other  situation 
would  have  a  significant  impact  on  the  acquisition.  An 
agree-disagree  a  Likert  scale  would  not  be  able  to 
discriminate  between  these  two  situations.  There  were  no 
scales  already  developed  that  would  directly  measure  impact. 
Therefore  the  scale  for  factor  one  was  developed  by  the 
researcher.  The  weakness  of  the  scale  is  that  it  has  not 
been  tested  and  its  reliability  may  be  suspect.  However,  it 
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is  the  only  scale  that  is  able  to  measure  the  data  of 
interest,  impact,  so  it's  validity  is  greater  than  any  other 
scale  considered.  The  scale  for  Factor  2  was  taken  from  the 
Army's  Questionnaire  Construction  Manual  (Questionnaire 
Construction  Manual,  1989:132). 

The  combination  of  these  two  factors,  impact  on  the 

program  and  frequency  of  occurrence,  were  used  together  to 

identify  pitfalls.  In  addition,  comments  were  solicited  on 

both  factors'  for  each  pitfall.  This  allowed  respondents  to 

provide  the  reasons  why  they  responded  the  way  they  did. 

According  to  Dillman,  allowing  respondents  to  create  their 

own  answers  is  most  often  used  when . . . 

...the  researcher  cannot  anticipate  the  various 
ways  in  which  people  are  likely  to  respond  to  a 
question.  They  are  used  to  stimulate  free 
thought,  solicit  suggestions,  probe  people's 
memories,  and  clarify  positions.  (Dillman, 

1978:87) 

This  open-ended  response  structure  was  used  to  provide 
feedback  to  respondents  in  the  subsequent  round;  thus 
allowing  respondents  to  consider  each  other's  informed 
opinion . 

Finally,  for  each  potential  pitfall,  the  survey 
solicited  an  open-ended  response  on  the  consequences  of  the 
pitfall  and  the  best  practice  a  source  selection  team  can 
employ  to  avoid  it . 
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The  survey  was  pre-tested  using  Source  Selection 
Officers.  The  purpose  of  pre-testing  this  group  was  to 
ensure  that  survey  questions  were  clear,  unbiased,  had 
understandable  vocabulary  and  allowed  adequate  alternatives 
(Emory  and  Cooper,  1991:362).  The  survey  was  also  pre¬ 
tested  using  AFIT  faculty  with  source  selection  experience. 
Recommendations  from  the  pretest  were  analyzed  and 
incorporated  into  the  survey  as  necessary.  After  this,  the 
first  round  survey  was  sent  to  the  survey  participants . 
Description  _af  Population 

A  population  is  the  total  number  of  subjects  that  meet 
the  target  characteristics  which  are  under  study  (Emory  and 
Cooper,  1991:245)  .  The  purpose  of  this  study  was  to 
identify  common  pitfalls  that  occur  in  source  selections  in 
the  four  AFMC  product  centers .  Further,  these  pitfalls 
would  be  the  ones  that  primarily  affected  the  contract 
def initization  team.  Therefore  the  population  is  all  source 
selections  conducted  in  all  of  the  product  centers. 

Description  of  Sample 

Because  source  selection  information  is  highly 
restricted,  it  would  be  impossible  to  evaluate  source 
selection  documentation  to  determine  if  common  pitfalls 
existed.  Further,  the  time  involved  to  study  many  source 
selections  in  depth  would  be  immense  and  costly.  To  get  at 
the  needed  information,  it  was  decided  to  elicit  the  opinion 
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of  source  selection  personnel  using  the  Delphi  method.  This 
method  works  best  if  the  respondents  used  are  experts  on  the 
topic  being  studied  (Brown :4) .  Therefore,  the  sample  used 
was  source  selection  experts  from  the  four  AFMC  product 
centers.  Since  these  experts  had  participated  in  numerous 
source  selections  in  each  of  the  four  centers,  they 
represented  a  sample  of  all  source  selections  conducted  by 
the  centers.  The  method  of  selection  was  judgmental 
sampling,  with  sample  members  being  selected  based  on  a 
criterion  (Emory  and  Cooper: 275) .  The  experts  sampled  were 
chosen  based  on  the  recommendation  of  the  Source  Selection 
Officers  (SSO)  at  each  product  center.  Each  SSO  was 
instructed  to  recommend  the  most  experienced  source 
selection  personnel  in  his/her  product  center.  This 
researcher  theorized  that  the  most  experienced  people  would 
have  had  the  most  exposure  to  the  various  problems 
encountered  in  a  source  selection  and  therefore  would  be 
best  suited  for  this  study.  The  selection  was  also  based  on 
a  quota,  with  the  sample  having  equal  representation  from 
each  product  center.  This  was  done  to  help  answer  the 
inter-product  center  research  objective. 
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Determining  Consensus 

Once  the  responses  from  the  first  round  were  received, 
each  situation  was  analyzed  to  determine  whether  or  not  a 
consensus  was  reached. 

There  has  been  an  ongoing  series  of  theses  at  the  Air 
Force  Institute  of  Technology  concerning  the  definition  of 
contracting  and  acquisition  terms.  Each  study  has 
investigated  a  different  set  a  acquisition  related  terms. 
These  studies  all  sought  consensus  when  testing  a  proposed 
definition  using  a  five  point  Likert  scale.  The  older 
theses  used  a  50%  response  rate  in  the  two  agreement 
response  categories  (Agree  and  Strongly  Agree)  to  conclude 
that  there  is  a  consensus  on  the  proposed  definition  (Moyle, 
1990:3-8;  Shelly,  1991:34;  others).  More  recent  studies  for 
contracting  terms,  however,  have  employed  a  more  restrictive 
decision  rule  of  66%  for  concluding  consensus  (Stormer  and 
Zigman,  1993 :A-6;  others).  According  to  Stormer  and  Zigman, 
if  exactly  50%  of  respondents  agree  with  a  definition,  then 
50%  of  the  respondents  disagree  also,  showing  a  complete 
lack  of  agreement  (Stormer  and  Zigman,  1993 :A-6). 

This  researcher  chose  to  use  a  more  restrictive 
response  rate  to  indicate  a  consensus.  If  at  least  two 
thirds  (66.6%)  of  the  responses  fell  into  two  adjacent 
categories,  this  was  an  indication  of  a  group,  consensus 
response.  Unlike  the  theses  on  contracting  terms,  this 
study  looked  for  a  consensus  on  the  entire  spectrum  of  both 
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scales;  Factors  1  and  2.  Thus,  if  two  thirds  of  the 
responses  fell  into  the  1  and  2  categories,  this  also  would 
indicate  a  consensus  response. 

A  consensus  of  the  consequences  and  best  practices  was 
not  sought  in  this  study  for  several  reasons.  First,  in 
round  one,  for  each  potential  pitfall,  respondents  described 
the  consequences  and  best  practices.  Thus,  possible 
consequences  and  best  practices  were  not  identified  until 
after  the  first  round.  To  find  a  consensus  on  the 
consequences’  and  best  practices,  a  third  round  questionnaire 
may  have  been  needed.  Time  constraints  did  not  allow  for 
the  development,  administration,  and  analysis  of  a  third 
round  survey.  Second,  for  each  pitfall  there  could  be 
several  consequences  and  best  practices.  This  would 
generate  a  number  of  new  questions  that  would  have  to  be 
added  to  the  survey,  making  it  extremely  long  (and 
potentially  confusing) .  This  violated  the  TDM  approach  to 
surveying  (Dillman,  1978:12,  14).  It  was  not  reasonable  to 
expect  the  survey  participants  to  complete  a  three  round 
survey  that  was  extremely  long.  Finally,  possible 
consequences  of  a  pitfall  are  most  likely  probabilistic. 
There  could  be  a  wide  range  of  possible  outcomes  if  a 
pitfall  is  encountered,  from  mild  to  severe.  Similarly, 
there  may  be  several  different  actions  a  source  selection 
team  could  take  to  avoid  a  pitfall.  Seeking  consensus  in 
this  situation  seemed  inappropriate.  For  purposes  of  this 
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study,  it  was  considered  more  valuable  and  reasonable  to 
describe  the  wide  range  of  possible  consequences  and 
available  best  practices . 


The  respondents  agreed  on  the  impact  and  frequency  on 
nine  of  the  fifteen  situations  in  the  first  round  survey. 

Of  the  six  situations  they  did  not  agree  on,  four  were  due 
to  divergent  opinions  on  the  impact  the  situation  would  have 
on  the  program  (Situations  2,  6,  8  and  12),  one  was  due  to 
divergent  opinions  on  the  frequency  with  which  the  situation 
occurred  (Situation  14),  and  one  received  divergent 
responses  on  both  impact  and  frequency  (Situation  5) . 

Additionally,  respondents  provided  the  consequences  of 
and  best  practices  to  avoid  each  situation.  These  responses 
will  be  discussed  in  detail  in  Chapters  IV  and  V. 


The  round  two  survey  re-tested  the  six  situations  that 
received  divergent  responses  in  the  first  round.  The  round 
two  survey  closely  resembled  the  round  one  survey.  The  same 
two  factors  (impact  and  frequency)  were  again  asked  of  each 
situation.  The  open-ended  questions  concerning  consequences 
and  best  practices  were  not  repeated  in  round  two.  This 
wasn't  necessary  as  this  information  was  already  collected 
in  round  one.  Additionally,  the  round  two  survey  included 
the  following  feedback  from  round  one  on  the  six  situations; 
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1 .  The  frequency  for  all  responses  for  each 
of  the  closed-ended  questions . 

2 .  The  mean  response  for  each  closed-ended 
question . 

3 .  The  response  the  individual  gave  to  each 
of  the  closed-ended  questions . 

4 .  All  write  in  comments  received  on  each 
question . 

Appendix  B  shows  the  feedback  given  with  the  round  two 
survey  while  Appendix  C  contains  the  round  two  survey. 

Survey  Results  -  Round  Two 

The  survey  participants  converged  in  their  responses  in 
the  second  round  on  the  remaining  six  situations.  All  six 
received  more  than  the  two-thirds  response  rate  (indicating 
consensus)  in  adjacent  categories  for  both  factors. 

The  final  results  received  for  Factors  1  and  2  are 
shown  for  all  situations  in  the  histograms  in  Appendix  D. 

For  situations  2,  5,  6,  8,  12  and  14  these  histograms 
reflect  round  two  results.  For  all  other  situations,  the 
histograms  are  based  on  round  one  data . 

Analysis  of  Data 

The  five  research  objectives  were  analyzed  for  each 
situation  in  the  following  way. 
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Research  Qfejectiye  1  -  Examine  the  extent  to  which 
source  selection  experts  can  agree  on  situations  that  could 
be  detrimental  to  a  source  selection. 

Each  factor  in  each  situation  was  analyzed  to  determine 
if  at  least  two  thirds  of  the  responses  fell  into  adjacent 
categories . 

Research  Objective  2  -  Examine  the  impact  these 
situations  have  on  the  acquisition  program  and  their 
frequencv  of  occurrence. 

Once  a  consensus  was  reached  on  a  situation  for  both 
factors,  the  central  tendency  of  the  responses  was  sought. 
The  central  tendency  was  analyzed  both  in  terms  of  median 
and  mean.  The  open  ended  responses  were  also  used  to 
evaluate  this  research  objective. 

Research  Objective  3  -  Examine  the  extent  to  which 
these  pitfalls  are  unique  to  particular  product  centers  or 
common  to  all  Air  Force  Material  Command  (AFMC)  product 
centers . 

For  each  factor,  the  medians  of  each  product  center 
was  compared  to  the  consensus  response  categories  on  each 
situation.  This  showed  how  each  product  center's  central 
tendency  compared  to  the  group  as  a  whole.  The  ranges  of 
each  product  center  were  also  used  to  compare  dispersion. 
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Determine  what  the 


consequences  are  to  the  acquisition  process  if  one  of  these 
pitfalls  is  encountered. 

The  responses  received  to  the  opened  ended  questions 
concerning  the  consequences  for  each  situation  were  used  to 
answer  this  research  objective. 

Research  Objective  5  -  Determine  the  extent  to  which 
Air  Force  source  selection  personnel  can  take  certain  steps 
(best  practices)  to  avoid  these  pitfalls  before  they  occur. 

The  responses  received  to  the  opened  ended  questions 
concerning  the  best  practices  used  to  avoid  each  situation 
were  used  to  investigate  this  research  objective . 


Summary 

This  study  investigated  the  pitfalls,  consequences  and 
best  practices  in  the  evaluation,  negotiation  and  award 
phases  of  a  source  selection  for  the  four  AFMC  product 
centers.  Because  of  the  lack  of  potential  pitfalls  in 
existing  literature,  a  two-stage  research  design  was  chosen. 
Stage  I  was  an  exploratory  phase  using  an  informal 
experience  survey.  It  polled  source  selection  experts  on 
their  views  concerning  pitfalls  in  the  source  selection 
process.  They  also  rendered  their  opinion  on  a  list  of 
possible  pitfalls  identified  by  the  researcher. 
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Stage  II  was  a  Delphi  survey  developed  using  the 
results  of  Stage  I  and  the  research  objectives.  The  survey 
was  conducted  in  two  rounds .  In  the  second  round,  the 
participants  received  feedback  from  the  first  round. 

Data  from  these  two  rounds  were  analyzed  to  see  if  the 
respondents  reached  a  consensus  on  any  of  the  possible 
pitfalls  and  to  identify  the  various  consequences  of  and 
best  practices  to  avoid  these  pitfalls.  The  results  of  this 
analysis  is  found  in  the  next  section.  Chapter  IV. 
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IV. 


This  chapter  describes  the  analysis  that  was  done  for 
this  study.  First,  the  survey  participants  are  discussed. 
Next,  each  situation  is  analyzed  individually  in  the  context 
of  the  survey  results  and  Research  Objectives  1,  2,  4  and  5. 
After  this  individual  analysis  of  each  situation,  an 
analysis  of  the  data  for  Research  Objective  3  is  presented. 

Description  of  Survey  Participants 

A  total  of  twenty  surveys  were  sent  out,  five  to  each 
of  the  four  product  centers .  Fourteen  surveys  were 
completed  and  returned  for  a  74%  response  rate.  Fourteen 
second  round  surveys  were  sent  out  to  the  people  that 
responded  in  the  first  round.  All  14  second  round  surveys 
were  completed  and  returned.  The  numbers  of  responses  by 
product  center  are;  ASC,  ESC  and  SMC,  three  each;  HSC, 
five.  The  rank  or  grade  of  ten  of  the  fourteen  participants 
was  either  GM-13  or  above  or  Colonel,  and  thus  indicated 
significant  civil  service  or  military  experience.  The 
participants  were  also  well  educated.  Most  (10  of  14)  had  a 
masters  degree  and  one  had  a  doctorate.  The  remaining  three 
held  a  bachelors'  degree.  Twelve  of  the  participants  had 
five  or  more  years  of  source  selection  experience.  Seven  of 
these  had  over  ten  years  of  source  selection  experience. 

Thus  the  sample  represented  people  with  a  fairly  high  degree 
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of  source  selection  experience.  Complete  background 
information  on  the  participants  is  provided  in  the  tables  in 
Appendix  E. 

Final.  _Raa.ul.tL3  -  Factors.  1  .and,  2 

Table  4.1  shows  the  final  results  received  on  all  the 
situations  for  Factor  1,  impact.  For  each  situation,  the 
table  shows  the  number  of  responses  received  for  each 
response  category.  For  example,  for  Situation  #1  no 
participants  chose  response  categories  1-None  or  2-Minor. 

Two  participants  chose  response  category  3-Moderate,  five 
chose  response  category  4-Significant  and  seven  participants 
chose  response  category  5-Very  Significant.  Response 
categories  4-Significant  and  5-Very  Significant  are 
highlighted  to  show  that  the  group  consensus  fell  into  these 
two  response  categories .  The  last  column  in  the  table  shows 
the  percentage  of  responses  in  these  two  categories  to  the 
total  responses  received.  For  Situation  #1,  the  consensus 
rate  is  85.7%  (12  responses  in  categories  4  and  5  divided  by 
the  14  responses  received) .  Some  situations  have  three 
response  categories  highlighted.  This  is  because  there  are 
two  possible  sets  of  consensus  categories  that  are  equal  to 
each  other.  In  Situation  #8,  for  example,  a  combination  of 
either  categories  3-Moderate  and  4-Significant  or 
4-Significant  and  5-Very  Significant  yields  a  response  rate 
of  78.6%.  Table  4.2  is  constructed  in  the  same  fashion  as 
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Table  4.1.  Table  4.2  shows  the  final  results  for  Factor  2, 
frequency. 

Both  tables  show  the  range  of  responses,  the  degree  of 
consensus,  the  most  frequent  response  and  the  central 
tendency  of  responses  for  each  situation.  The  information 
in  these  tables  will  be  referred  to  as  necessary  during  the 
analysis  of  each  situation. 


Table  4.1  -  Final  Results  for  Factor  1,  Impact 


\  * 

Situation  Description 

RMpOU* 

Categories* 

1  2  3  4  5 

%  in 

consensus 

categories 

1 

Standards  not  in  RFP 

El 

El 

2 

B 

B 

85.7 

> 

2 

Strengths  and  weaknesses  rated 
differently 

0 

H 

6 

B 

B 

92.9 

3 

Evaluators  lack  of  global  view 

0 

II 

5 

5 

0 

71.4 

4 

Redundant  CR/DRs 

n 

B 

B 

B 

78.6 

5 

CR/DR  rewrites  change  meaning 

B 

B 

B 

B 

B 

71.4 

6 

All  costs  unreasonable  using 
analysis  technique 

0 

i 

2 

B 

B 

78.6 

■ 

Hidden  past  performance 
problems 

2 

2 

B 

B 

B 

69.2 

8 

Sticking  by  "award  without 
discussions"  policy 

0 

H 

3 

B 

3 

78.6 

9 

Eliminate  offers  in 
competitive  range  to  reduce 
number 

H 

B 

2 

6 

R 

71.4 

10 

Discussing  issues  not 
addressed  by  a  CR/DR/MR 

0 

H 

B 

2 

B 

mm 

11 

Technical  leveling, 
transfusion,  auctioning 

0 

B 

B 

B 

B 

92.3 

12 

Not  all  deficiencies  are 
discussed  with  offerors 

0 

B 

3 

5 

6 

78.6 

13 

BAFO  prices  exceed  funds 
available 

0 

B 

fl 

6 

6 

92.3 

14 

Bringing  new  people  into  the 
evaluation/negotiation  phase 

0 

5 

B 

2 

B 

85.7 

15 

Arbitrary  schedule 

0 

1 

5 

~ 

85.7 

♦Factor  1  scale  (Impact) :  1-None,  2-Minor,  3-Moderate, 
4-Significant,  5-Very  Significant 
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Table  4.2  -  Final  Results  for  Factor  2,  Frequency 


§ 

Situation  Description 

Response 

Categories* 

1  2  3  4  5 

%  in 

consensus 

categories 

1 

Standards  not  in  RFP 

b 

B 

B 

1 

0 

78.6 

2 

Strengths  and  weaknesses  rated 
differently 

0 

H 

H 

n 

0 

92.9 

3 

Evaluators  lack  of  global  view 

O 

B 

B 

B 

1 

79.6 

4 

Redundant  CR/DRs 

0 

O 

B 

LL 

1 

71.4 

5 

CR/DR  rewrites  change  meaning 

m 

B 

B 

B 

1 

71.4 

6 

All  costs  unreasonable  using 
analysis  technique 

H 

H 

n 

2 

0 

78.6 

fl 

Hidden  past  performance 
problems 

3 

H 

5 

n 

0 

69.2 

8 

Sticking  by  "award  without 
discussions"  policy 

3 

II 

H 

0 

0 

78.6 

9 

Eliminate  offers  in 
competitive  range  to  reduce 
number 

6 

5 

1 

2 

0 

78.6 

10 

Discussing  issues  not 
addressed  by  a  CR/DR/MR 

2 

4 

5 

2 

0 

69.2 

11 

Technical  leveling, 
transfusion,  auctioning 

H 

H 

n 

B 

0 

92.3 

12 

Not  all  deficiencies  are 
discussed  with  o.fferors 

2 

H 

H 

B 

0 

78.6 

13 

BAFO  prices  exceed  funds 
available • 

2 

6 

H 

0 

0 

83.3 

14 

Bringing  new  people  into  the 
evaluation/negotiation  phase 

0 

0 

3 

8 

3 

78.6 

15 

Arbitrary  schedule 

0 

0 

5 

8 

1 

92.9 

♦Factor  2  scale  (Frequency) :  1 -Never,  2-Seldom,  3-SometiiAes, 

4-Usually,  5-Always 


Research  -Qfcgggtigsa 

The  data  received  on  each  situation  was  analyzed  for 
each  research  objective.  Research  Objective  1  concerned  the 
extent  to  which  the  survey  participants  could  agree  on  the 
situation  tested.  Research  Objective  2  looked  at  the  impact 
and  frequency  of  these  situations .  Research  Objectives  4 


4-4 


and  5  focused  on  the  consequences  and  best  practices  of  each 
situation  respectively.  Below  are  the  results  of  this 
analysis  for  each  situation.  As  stated  previously.  Research 
Objective  3  will  be  discussed  after  the  analysis  of  each 
situations  for  the  other  four  research  objectives . 

Analysis  of  Situation  #1 

Situation  #1  was  "A  technical  team  membe'r  evaluates  a 
proposal  based  on  standards  not  in  the  Request  for  Proposal 
(RFP) ." 

Discussion :  A  consensus  on  impact  and  frequency  was 
reached  on  this  situation  in  the  first  round.  Tables  4.1 
and  4.2  show  that  the  extent  of  consensus  for  this  situation 
was  85.7%  for  impact  and  78.6%  for  frequency.  Most 
participants  stated  that  this  was  a  significant  problem  and 
sometimes  did  occur.  Generally,  differences  of  opinion  on 
impact  was  attributed  to  when  the  error  was  discovered.  If 
the  SSET  found  the  discrepancy  then  the  evaluation  could  be 
corrected.  If  it  was  discovered  after  award  during  the 
debriefing  session  with  the  unsuccessful  contractor,  then 
the  potential  for  a  protest  would  be  very  high.  Because  the 
impact  was  severe  if  the  situation  was  discovered  after 
award,  many  participants  said  they  had  never  seen  this 
happen.  However,  even  if  caught  before  contract  award, 
correction  of  the  improper  evaluation  would  take  "much  time 
and  effort",  and  thus  would  still  be  a  problem.  As  one 
expert  said,  " [We]  continuously  have  team  members  who  are 
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unfamiliar  with  source  selections . "  Responses  on  the  best 
practices  to  avoid  this  situation  included  training 
technical  team  members  before  evaluations  begin.  Also,  the 
contracting  officer  and  SSET  chairperson  should  monitor 
evaluations  early  and  be  prepared  to  correct  this  problem 
immediately  should  it  occur. 

Analysis  of  Situation  #2 

Situation  #2  was  "Similar  strengths  and  weaknesses 
between  proposals  get  rated  differently." 

Discussion :  A  consensus  was  not  reached  on  the  impact 
of  this  situation  in  the  first  round  (64.3%) .  A  consensus 
was  reached  in  the  second  round.  This  situation  had  the 
highest  rate  of  consensus  for  both  factors  of  all  the 
situations  tested.  The  rate  was  92.9%  for  both  impact  and 
frequency.  Therefore,  there  was  a  very  high  degree  of 
agreement  among  the  participants  on  this  situation.  The 
impact  of  this  situation  was  between  "moderate"  and 
"significant."  The  frequency  with  which  this  situation 
occurred  was  between  "seldom"  and  "sometimes."  The  comments 
received  on  this  situation  revealed  that  the  impact  of  this 
situation  depended  on  two  things;  when  it  was  caught  and 
whether  or  not  the  part  of  the  proposal  that  was  rated 
differently  was  critical  to  the  final  selection.  If  the 
error  occurred  on  a  relatively  minor  evaluation  factor,  its 
impact  would  be  relatively  minor.  One  respondent  indicated 
that  the  impact  of  this  error  could  also  vary  depending  on 
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each  offeror's  individual  approach.  If,  for  example,  an 
offeror  receives  an  unusually  low  score  on  his  proposed 
subcontracting  plan,  the  effect  this  would  have  on  his 
chances  of  winning  the  contract  would  depend  on  how  much  of 
the  work  he  has  proposed  to  subcontract.  The  more  work  he 
proposed  to  contract  out,  the  more  impact  this  rating  would 
have  on  the  government's  evaluation  of  his  proposal.  If,  on 
the  other  hand,  he  intends  to  accomplish  most  of  the  work  in 
house,  the  perceived  weakness  in  his  subcontracting  plan 
(relative  to  similar  offers)  would  have  a  smaller  impact  on 
his  proposal.  The  consequences  of  this  situation  could 
include  a  sustained  protest  if  the  different  ratings 
affected  the  final  selection.  Best  practices  included 
having  one  person  evaluate  or  at  least  review  all  proposals 
for  the  same  factor  to  ensure  consistency. 

Analysis.  gf_  Situation-il 

Situation  #3  was  "Evaluators  lack  global  view  of 
proposal  and  do  not  pick  up  conflicting  or  clarifying 
information . ” 

Discussion :  A  consensus  on  impact  and  frequency  was 
reached  on  this  situation  in  the  first  round.  The  consensus 
on  impact  for  this  situation  was  among  the  four  lowest  rates 
of  all  the  situations  tested  at  71.4%.  The  comments 
received  on  this  pitfall  seemed  to  indicate  that  the 
respondents  were  somewhat  divided.  Several  considered  this 
a  minor  problem  that  would  (and  should)  get  fixed  at  a 
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higher  level.  As  one  respondent  said,  "As  areas  are  rolled 
up  to  form  an  overall  assessment  these  differences  will 
surface."  Another  group  thought  that  this  could  pose  a 
problem  to  the  source  selection.  Further,  they  felt  action 
at  the  evaluator  level  was  needed  to  correct  it.  According 
to  one  SMC  respondent,  "At  SMC  evaluators  are  required  to 
read  the  entire  proposal."  Overall  the  impact  was  rated  as 
"moderate."  SMC  respondents  all  rated  the  impact  as 
"significant"  while  ASC  and  ESC  respondents  leaned  toward  a 
"minor"  response.  The  frequency  of  this  situation  was  rated 
as  "sometimes"  by  most  respondents.  Possible  consequences 
of  this  situation  included  post  award  problems  such  as 
disputed  contractual  requirements  and  increased 
modifications.  Comments  on  the  best  practices  were  to 
either  have  evaluators  read  all  proposals  or,  if  this  was 
not  feasible,  to  have  frequent  meetings  between  teams  to 
review  and  share  information. 


Situation  #4  was  "High  number  of  redundant  deficiency 
reports  and  clarification  requests  (CR/DR)  are  generated." 

Discussion :  A  consensus  on  impact  and  frequency  was 
reached  on  this  situation  in  the  first  round.  This 
situation  had  a  consensus  rate  of  71.4%  on  frequency.  This 
was  among  the  four  lowest  consensus  rates  received  for  this 
factor.  Some  notable  comments  concerned  the  impact  this 
situation  has  if  encountered.  Some  experts  saw  this  as  a 
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positive  sign.  To  them  it  meant  that  the  evaluators  were 
thoroughly  reading  the  proposal  and  that  they  were  being 
consistent  in  their  evaluations.  Further,  they  indicated 
that  these  redundancies  could  be  consolidated  at  a  higher 
level.  This  situation  also  ranked  very  low  on  the  impact 
scale.  On  the  basis  of  these  results  this  situation  cannot 
be  considered  a  pitfall. 

Analysis of  Situation  »5 

Situation  #5  was  "Numerous  rewrites  of  a  DR/CR/MR 
before  it  is  approved,  sometimes  changing  its  original 
meaning . " 

Discussion:  •  This  was  the  only  situation  tested  that 
failed  to  get  a  consensus  response  on  both  impact  and 
frequency.  The  first  round  results  for  impact  and  frequency 
were  64.3%  and  57.1%  respectively.  A  consensus  on  these  two 
factors  was  reached  in  the  second  round,  although  it  was 
still  somewhat  low  relative  to  the  other  consensus  rates. 

The  rate  for  both  factors  was  71.4%  in  the  second  round. 

There  were  two  confusing  issues  that  may  have  led  to 
neither  factor  receiving  a  consensus  response.  First,  some 
respondents  did  not  understand  who  was  doing  the  rewriting 
in  the  scenario,  the  government  or  the  contractor.  They 
felt  that  this  could  mean  that  the  contractors  themselves 
were  responding  to  CR/DRs  and  had  to  rewrite  the  response 
several  times  before  their  response  was  accepted.  This  was 
not  the  intended  interpretation  of  the  situation.  The 


4-9 


rewriting  in  the  scenario  was  being  done  by  government 
personnel  before  the  CR/DR  was  approved  and  sent  to  the 
contractor . 

The  second  confusing  issue  concerned  the  use  of  the 
term  modification  request  (MR) .  Several  respondents  said 
that  modification  requests  (MRs)  did  not  fall  into  the  same 
categories  as  clarification  requests  (CRs)  and  deficiency 
reports  (DRs) .  The  term  MR  refers  to  an  RFP  amendment.  An 
amendment  would  not  be  issued  using  a  process  similar  to 
DR/CR  generation  so  these  comments  were  valid. 

As  a  result  of  these  two  finding  this  situation  was 
reworded  in  the  second  round  survey  to  read;  "Numerous 
rewrites  of  a  DR/CR  during  the  government  review  process, 
before  it  is  approved  and  issued  to  the  offeror,  sometimes 
changing  its  original  meaning."  Also,  the  comment  about  MRs 
for  this  scenario  is  also  applicable  to  Situation  #4. 

Most  comments  indicated  that  if  this  situation  occurred 
that  further  discussions  would  be  needed  to  address  the 
original  problem.  This  would  cause  unnecessary  delays. 

Best  practices  included  giving  evaluators  training  on  how  to 
write  CR/DRs .  Also,  if  a  re-write  is  necessary,  the 
evaluator  should  participate  in  or  review  the  re-write  to 
ensure  its  meaning  hasn't  changed. 
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Situation  #6  was  "The  price  or  cost  analysis  technique 
developed  for  the  cost/price  evaluation  renders  all  or  most 
offers  unrealistic  and  unreasonable  when  applied." 

Discussion :  A  consensus  was  not  reached  on  the  impact 
of  this  situation  in  the  first  round  (64.3%) .  It  was  in  the 
second  round.  Several  people  felt  that  if  this  situation 
were  encountered  it  could  be  fixed.  May  other  respondents, 
however,  indicated  that  the  RFP  would  have  to  be  canceled  if 
this  happened.  This  is  clearly  evident  as  the  most  frequent 
response  for  impact  was  "very  significant."  Respondents 
also  indicated  that  this  situation  rarely  happens.  The 
consequences  of  this  situation  ranged  from  amending  the  RFP 
to  canceling  the  solicitation.  Best  practices  include  using 
valid  cost/price  techniques  to  develop  the  government 
estimate.  It  was  also  important  to  use  draft  RFPs  and  pro¬ 
solicitation  conferences  to  ensure  the  final  RFP  is  clear. 

Analysis  of  Situation.- #Z 

Situation  #7  was  "Past  performance  problems  which, 
though  relevant  to  the  source  selection,  are  unknown  to  or 
hidden  from  the  Program  Risk  Assessment  Group  (PRAG) ." 

Discussion :  A  consensus  on  impact  and  frequency  was 
reached  on  this  situation  in  the  first  round.  This 
situation  had  the  lowest  combined  consensus  responses  of  any 
situation  tested.  The  consensus  response  rate  was  69.2%  for 
both  factors .  This  indicates  that  while  a  consensus  was 
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reached,  it  was  a  weak  consensus  with  some  divergent 
opinions.  Different  opinions  on  the  impact  of  this 
situation  seemed  to  depend  on  whether  the  respondent  was 
talking  about  the  impact  on  the  acquisition  program  or  the 
source  selection  process.  Most  that  referred  to  the  impact 
on  the  acquisition  program  thought  that  this  situation  could 
have  a  significant  impact.  Those  referring  to  the  impact  on 
the  source  selection  process  indicated  that  this  situation 
would  be  minor.  In  effect,  they  said  if  the  unfavorable 
information  was  unknown,  it  would  have  no  bearing  (and  thus 
no  impact)  on  the  source  selection.  The  acquisition 
program,  however,  would  be  affected  because  its  success 
would  be  linked  to  a  contractor  with  undesirable  performance 
characteristics.  One  respondent  summed  up  both  of  these 
perspectives .  "While  impact  on  the  source  selection  may  be 
minor,  problems  would  likely  arise  during  the  performance  on 
the  contract."  The  best  practices  to  avoid  this  situation 
included  a  suggestion  to  develop  acquisition  unique 
questionnaires  to  ensure  that  relevant  information  is 
gathered. 

Analysis.  jaf-Sitaatian  #9 

Situation  #8  was  "Sticking  by  an  'award  without 
discussions'  policy  when  it  is  found  that  discussions  are 
probably  warranted." 

Discussion :  A  consensus  was  not  reached  on  the  impact 

of  this  situation  in  the  first  round  (61.5%) .  It  was 
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reached  in  the  second.  Most  of  the  comments  received  for 
this  situation  were  straight  forward.  One  respondent 
indicated  that  this  can  become  a  "problem  when  [source 
selection  personnel]  are  intimidated  by  pressure  from  the 
top  to  award."  Surprisingly,  seven  respondents  said  that 
this  situation  occurs  "sometimes."  Very  few  of  the  comments 
concerned  the  possibility  of  a  protest  in  this  situation. 
Most  addressed  post  award  problems  such  as  misunderstandings 
of  the  requirement,  schedule  delays  and  changes.  Best 
practices  included  allowing  sufficient  time  for  discussions 
in  the  schedule.  This  was  extremely  interesting  as 
arbitrarily  determined  source  selection  schedules  was  a 
situation  tested  in  this  study  (Situation  #15) .  It  was  also 
found  to  occur  quite  often. 

Analysis  Qf.. -Situation,,  &i£ 

Situation  #9  was  "The  motivation  during  the  competitive 
range  determination  is  to  reduce  the  number  of  offerors,  not 
to  retain  offerors  whose  proposals  have  a  reasonable  chance 
of  being  selected  for  award." 

Discussion :  A  consensus  on  impact  and  frequency  was 
reached  on  this  situation  in  the  first  round.  The  consensus 
response  on  impact  was  among  the  four  lowest  rates  for  this 
factor  out  of  all  the  situations  tested.  It  was  71.4%. 

Also,  it  was  the  only  situation  in  which  the  responses  for 
impact  ranged  from  "none"  to  "very  significant."  One 
comment  from  SMC  indicated  that  because  they  usually  receive 
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only  two  to  three  proposals,  this  situation  does  not  have  a 
chance  to  occur.  Most  respondents  thought  a  situation  just 
the  opposite  from  the  one  tested  is  more  likely  to  happen. 
They  said  that  offerors  are  more  likely  to  be  left  in  the 
competitive  range  when  they  should  be  eliminated.  This 
situation  received  the  greatest  amount  of  responses  in  the 
"never"  category  of  all  the  situations  tested.  Six 
respondents  said  that  the  situation  tested  "never"  occurs. 
HSC  differed  from  the  rest  of  the  group  as  three  of  the  five 
HSC  respondents  said  that  this  situation  "sometimes"  or 
"usually"  occurs .  The  consequences  if  this  happens  is  that 
the  offeror  who  was  excluded  from  the  competitive  range  may 
file  and  win  a  protest.  Best  practices  included  having  the 
competitive  range  determination  reviewed  by  a  government 
contracts  attorney. 

Analysis _ of  ..Sitnati-an ... #.10. 

Situation  #10  was  "Addressing  issues  during  face  to 
face  discussions  which  were  not  addressed  with  a  DR/CR/MR." 

Discussion :  A  consensus  on  impact  and  frequency  was 

reached  on  this  situation  in  the  first  round.  The  impact 
this  situation  had  on  the  program  depended  on  the  nature  of 
the  fact-to-face  discussions.  Some  saw  these  additional 
discussions  as  necessary  if  they  brought  to  light  a 
deficiency  not  covered  by  a  CR/DR.  Although  this  may  create 
delays,  it  is  "better  late  than  never."  A  few  other 
respondents  indicated  that  additional  discussions  could  lead 
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to  problems  such  as  technical  leveling.  Overall,  most 
respondents  found  this  to  be  only  a  minor  problem.  This 
situation  tested  lower  than  any  other  on  the  impact  scale. 
They  also  indicated  it  did  not  happen  very  often.  Based  on 
this.  Situation  #10  is  not  considered  a  pitfall. 

Analysis  of  SituaLi.Qn_i.il 

Situation  #11  was  "Technical  leveling,  transfusion  or 
auctioning  during  negotiations." 

Discussion :  A  consensus  on  impact  and  frequency  was 
reached  on  this  situation  in  the  first  round.  This 
situation  had  a  high  consensus  rate  on  both  impact  and 
frequency,  both  92.3%.  Most  respondents  found  this 
situation  would  have  a  very  significant  impact  on  the 
program.  Further,  most  said  it  will  occur  very 
infrequently.  This  is  probably  because  technical  leveling, 
transfusion  and  auctioning  are  explicitly  prohibited  by  the 
FAR.  An  examination  of  the  comments  revealed  an  interesting 
insight.  Several  respondents  said  that  technical  leveling 
may  occur  as  proposals  are  evaluated  and  may  lead  to 
technical  transfusion.  Auctioning  is  not  significantly 
related  to  the  other  two  actions.  The  frequency  with  which 
these  actions  occur  may  not  be  the  same  for  all  three 
actions.  This  evidence  suggests  that  there  are  multiple 
issues  within  this  situation.  Based  on  this  data,  this 
situation  should  be  re-tested  as  three  separate  situations. 
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The  situation  as  written  will  not  be  included  as  a  pitfall 
in  this  study. 

Analysis  gf  Situation..  112 

Situation  #12  was  "Not  all  proposal  deficiencies  are 
not  addressed  during  discussions  with  offerors." 

Discussion :  A  consensus  was  not  reached  on  the  impact 

of  this  situation  in  the  first  round  (61.5%).  It  was 
reached  in  the  second  round.  After  receiving  feedback  on 
this  pitfall  in  the  first  round,  there  was  a  3hift  in 
responses  towards  rating  the  impact  of  this  situation  more 
significantly.  The  response  categories  "significant"  and 
"very  significant"  increased  by  four  responses  in  the  second 
round  and  the  categories  "minor"  and  "moderate"  decreased  by 
three  responses.  The  consequences  to  this  situation 
included  having  a  protest  sustained  by  an  offeror  who  was 
not  alerted  to  a  proposal  deficiency.  Best  practices 
include  comparing  identified  weaknesses  in  a  proposal  to  the 
CR/DRs  that  have  been  prepared.  This  will  ensure  that  the 
offeror  will  be  notified  of  any  deficiencies.  Another  best 
practice  would  be  to  limit  the  areas  to  be  evaluated  to 
those  things  that  are  critical  to  performance.  This  will 
reduce  proposal  size  and,  consequently,  the  likelihood  of 
overlooking  a  deficiency  during  the  evaluations  will  be 


reduced . 


Analysis  of  Situation  #13 

Situation  #13  was  "BAFO  prices  exceed  funds  available." 

Discussion ;  A  consensus  on  impact  and  frequency  was 
reached  on  this  situation  in  the  first  round.  There  was  a 
high  degree  of  consensus  on  the  impact  of  this  situation, 
92.3%.  The  consensus  response  on  impact  was  from 
"significant"  to  "very  significant."  Most  respondents  said 
that  this  situation  "seldom"  occurs.  Most  of  the 
consequences  and  best  practices  were  very  consistent  for 
this  situation.  If  this  situation  occurs  many  respondents 
said  that  the  solicitation  would  have  to  be  amended  or 
canceled.  Best  practices  to  avoid  this  situation  is  to 
share  the  funding  profile  with  offerors  when  funding  is 
constrained. 


Analysis  s£  Situation  #14 

Situation  #14  was  "Bringing  people  into  the 
evaluation/negotiation  phase  of  the  source  selection  who 
were  not  involved  in  the  development  of  the  RFP  and  award 
criteria . " 

Discussion :  A  consensus  was  not  reached  on  the 
frequency  that  this  situation  occurs  in  the  first  round 
(64.3%).  It  was  reached  in  the  second.  Some  respondents 
said  that  adding  people  during  the  evaluation  phase  is 
natural,  especially  if  a  high  number  of  proposals  are 
received.  Also  noteworthy,  many  of  the  responses  indicated 
that  source  selection  managers  do  not  have  a  great  deal  of 
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control  over  personnel  issues  3uch  as  this.  Most 
respondents  said  this  situation  would  have  a  "minor"  to 
"moderate"  impact  on  the  program.  Mso,  it  appears  that 
this  situation  is  the  norm  for  most  source  selections.  Most 
respondents  said  that  this  "usually"  occurs .  The  most 
likely  consequence  would  be  a  delay  in  the  source  selection 
as  new  people  are  brought  up  to  speed.  Although  source 
selection  managers  may  not  have  complete  control  over  this 
situation,  the  stability  of  personnel  can  be  evaluated  when 
selection  source  selection  am  members. 


Analysis  of  Situation  #15 

Situation  #15  was  "Trying  to  accomplish  the  source 
selection  within  an  arbitrarily  determined  schedule." 

Discussion :  A  consensus  on  impact  and  frequency  was 

reached  on  this  situation  in  the  first  round.  This 
situation  equaled  Situation  #2  for  the  highest  consensus 
response  rate  on  frequency  of  92.9%.  This  situation  had  the 
highest  combined  scores  on  the  impact  and  frequency  of  all 
the  situations  tested.  In  other  words,  it  was  the  situation 
with  the  highest  impact  on  a  program  that  happened 
relatively  often.  The  consensus  response  was  between 
"moderate"  and  "significant"  for  impact  and  between 
"sometimes"  and  "usually"  for  frequency.  Ironically,  a  best 
practice  to  avoid  Situation  #8,  which  was  awarding  without 
discussions,  was  to  allow  for  discussions  when  developing 
the  schedule.  Yet  it  appears  that  arbitrarily  determined 
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source  selection  schedules  are  the  norm.  It  is  possible 
that  arbitrary  schedules  precipitate  the  decision  to  award 
without  discussions,  even  when  they  are  needed.  If,  on  the 
other  hand,  proper  source  selection  procedures  are  followed 
and  the  schedule  slips,  the  user  may  become  upset.  The  best 
practices  to  avoid  this  situation  include  allowing  schedule 
time  for  contingencies  when  developing  the  source  selection 
schedule . 


Analysis  of  Research  Objective  3 

The  low  number  of  respondents  from  each  of  the  four 
product  centers  (three  each  from  A SC,  ESC,  and  SMC;  five 
from  H SC)  makes  this  research  objective  difficult  to  answer. 
Recognizing  this  limitation  and  acknowledging  that  further 
research  is  needed,  the  data  did  indicate  that  the  frequency 
and  impact  may  be  different  for  some  product  centers  than 
for  AFMC  as  a  whole.  Tables  4.3  and  4.4  illustrate  these 
possible  differences.  In  Chapter  III,  a  consensus  response 
was  defined  as  a  factor  that  received  at  least  two  thirds  of 
the  total  responses  in  adjacent  categories.  For  each 
situation,  the  tables  show  in  what  adjacent  categories  this 
consensus  was  located.  The  tables  also  show  what  the  median 
response  was  for  each  product  center . 

For  example,  in  Table  4.3,  Situation  #3  shows  that 
the  median  responses  by  product  center  were,  2-Minor  for  ASC 
and  ESC,  3-Moderate  for  HSC  and  4-Significant  for  SMC.  The 
consensus  response  categories  were  3-Moderate 
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Table  4.3  -  Comparison  of  Product  Center  Median 
Responses  and  Ranges  for  Factor  1,  Impact 
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^Factor  1  scale  (Impact) :  1-None,  2-Minor,  3-Moderate,  4-Significant,  5-Very  Significant 


Table  4.4  -  Comparison  of  Product  Center  Median 
Responses  and  Ranges  for  Factor  2,  Frequency 
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♦Factor  2  scale  (Frequency) :  1-Never,  2-Seldom,  3-Sometimes,  4-Usually,  5-Always 


and  4-Significant.  These  were  the  two  categories  that 
contained  at  least  66.6%  of  the  total  responses.  The  median 
responses  for  ASC  and  ESC  are  highlighted  because  their 
median  responses  were  not  in  the  consensus  response 
categories .  This  analysis  compares  the  central  tendency  of 
each  product  center  to  the  central  tendency  of  the  group  as 
a  whole. 

The  tables  also  gives  the  range  of  responses  for  each 
product  center  as  well  as  the  group  range.  Looking  back  at 
Table  4.3,  Situation  #3,  the  range  of  responses  for  ASC  is 
1 .  This  range  was  calculated  by  subtracting  the  lowest 
response  received  by  an  ASC  respondent  by  the  highest 
response  received  by  an  ASC  respondent.  For  Situation  #3, 
Factor  1,  the  lowest  ASC  response  was  2 -Minor  and  the 
highest  ASC  response  was  3-Moderate.  The  difference  between 
these  two  responses  is  1 .  The  product  center  ranges  assess 
the  amount  of  dispersion  that  is  present  within  each  product 
center.  Table  4.3  contains  median  and  range  data  for  Factor 
1,  impact.  Table  4.4  contains  the  same  type  of  data  for 
Factor  2,  frequency. 

To  assess  whether  or  not  a  situation  was  different 
for  a  product  center  for  either  impact  or  frequency  from  the 
group  as  a  whole,  a  combination  of  central  tendency  and 
range  were  both  analyzed.  The  central  tendency  was  first 
assessed  to  see  if  it  fell  outside  the  consensus  response 
categories.  Next,  the  range  was  analyzed  for  those  product 
centers  whose  median  was  found  to  be  outside  the  consensus 
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categories.  A  low  range  (range  of  0  or  1)  indicated  that 
respondents  from  the  same  product  center  were  in  general 
agreement  among  themselves . 

For  example,  for  Situation  #3,  Factor  1  in  Table  4.3, 
the  ESC  median  response  falls  outside  the  consensus 
categories.  The  range  for  ESC  on  this  situation  is  low. 
Thus,  not  only  is  the  ESC  response  outside  the  consensus 
range,  but  there  is  little  dispersion  among  the  ESC 
respondents  for  this  response .  Contrast  this  example  with 
Situation  #5,  Factor  1  in  the  same  table.  The  ESC  median 
here  is  also  not  in  the  consensus  categories .  The  range, 
however,  is  high  and  indicates  that  the  agreement  between 
the  ESC  participants  on  the  impact  of  this  situation  is  low. 
Because  of  this,  it  is  not  clear  if  the  impact  of  this 
situation  on  ESC  is  really  different  than  the  impact  on  the 
entire  group. 

Product  centers  whose  medians  vrere  not  in  the 
consensus  range  and  whose  range  was  low  were  judged  to  have 
a  different  response  than  the  group  response.  Based  on  this 
criteria,  it  appears  that  the  following  product  center 
responses  differ  from  the  group  response: 

Facto-r  1  (.Impact  1.  Pif ffixfinc.&a 

Product  Centers 
Consensus  with  Different 

Situation  #  Categories  Response 

3.  Evaluators  lack  global  3-Moderate  ASC  (2-Minor) 

view.  4-Significant  ESC  (2-Minor) 
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Factor  2  (Frequency)  Differences 


Situation  # 

Consensus 

Categories 

Product  Center 
with  Different 

Median  Response 

4.  Redundant  CR/DRs 

2- Seldom 

3- Sometimes 

ESC 

(4-Usually ) 

7.  Hidden  poor  past 

performance  problems 

2- Seldom 

3- Sometimes 

ASC 

(Between  1-Never 
and  2-Seldom) 

8 .  Sticking  by  "award 
without  discussions" 
policy 

2- Seldom 

3- Sometimes 

ESC 

(1-Never) 

13 .  BAFO  prices  exceed 
funds  available 

2- Seldom 

3- Sometimes 

ASC 

(Between  1-Never 
and  2-Seldom) 

It  is  important  to  re-state  the  serious  limitation  on 
this  conclusion.  For  ASC,  ESC  and  HSC,  the  three  product 
centers  with  only  three  respondents,  a  change  in  response  by 
just  one  respondent  could  move  that  product  center's  median 
into  or  out  of  a  consensus  response  category. 

Summary 

This  chapter  provided  background  information  on  the 
survey  participants  and  provided  descriptive  statistics  on 
the  data  collected.  It  also  analyzed  each  situation  based 
on  the  research  objectives .  Finally,  it  analyzed  Research 
Objective  3  separately. 

The  next  chapter  will  draw  several  conclusions  and 
recommendations  based  on  this  study.  It  will  also  provide 
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a  template  of  the  consequences  and  best  practices  for  the 
pitfalls  identified  in  this  study.  Finally,  research 
limitations  and  areas  for  further  research  will  be 
discussed . 
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V. 


QgfijaEiSH 

This  chapter  draws  conclusions  as  a  result  of  this 
study.  First,  it  will  use  the  data  and  data  analysis  to 
make  some  concluding  remarks  on  the  research  objectives . 
Next,  some  general  conclusions  will  be  presented.  Finally, 
recommendations,  limitations  of  this  study,  and  possible 
areas  for  future  research  will  be  discussed. 

Research  Objective  1  -  Examine  the  extent  to  which  source 
selection  experts  can  agree  on  situations  that  could  be 
detrimental , to  a .source  selection: 

As  stated  previously,  there  were  15  situations  that 
were  tested  as  potential  pitfalls .  The  survey  participants 
agreed  on  the  impact  and  frequency  of  all  15  situations . 
Nine  of  the  15  situations  were  agreed  upon  in  the  first 
round,  with  no  feedback  necessary  for  the  participants  to 
reach  a  consensus.  Six  of  the  situations  did  not  achieve  a 
consensus  response  in  the  first  round.  Table  5.1  shows  the 
first  round  results. 
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Table  5.1  -  Extent  of  Consensus  of 
Round  One  for  Factors  1  and  2 


i 

Situation  Description 

Percent  of  total 
responses  in  adjacent 
categories 

Factor  1* 

Factor  2** 

mm 

Standards  not  in  RFP 

85.7% 

78.6 

2 

Strengths  and  weaknesses  rated 
differently 

64.3% 

85.7 

3 

Evaluators  lack  of  global  view 

71.4% 

79.6% 

mm 

Redundant  CR/DRs 

78.6% 

71.4% 

5 

CR/DR  rewrites  change  meaning 

64.3% 

57.1% 

6 

All  costs  unreasonable  using 
analysis  technique 

64.3% 

78.6% 

7 

Hidden  past  performance  problems 

69.2% 

69.2% 

8 

Sticking  by  "award  without 
discussions"  policy 

61.5% 

76.9% 

9 

Eliminate  offers  in  competitive 
range  to  reduce  number 

71.4% 

78.6% 

10 

Discussing  issues  not  addressed  by 
a  CR/DR/MR 

76.9% 

69.2% 

11 

Technical  leveling,  transfusion, 
auctioning 

92.3% 

92.3% 

12 

Not  all  deficiencies  are  discussed 
with  offerors 

61.5% 

76.9% 

13 

BAFO  prices  exceed  funds  available 

92.3% 

83.3% 

14 

Bringing  new  people  into  the 
evaluation/negotiation  phase 

92.9% 

64.3% 

15 

Arbitrary  schedule 

85.7% 

92.9% 

*Factor  1  -  Impact  situation  has  on  the  acquisition 
program . 

**Factor  2  -  Frequency  that  situation  occurs  during  source 
selections . 


The  six  situations  that  did  not  have  a  consensus 
response  on  either  impact,  or  frequency  or  both  are 
highlighted  in  Table  5.1.  These  six  situations  were  re¬ 
tested  again  in  the  second  round  survey.  A  consensus 
response  was  achieved  on  these  six  situations  in  the  second 
round.  The  extent  that  a  consensus  was  reached  in  the 
second  round  is  shown  in  Table  5.2. 
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Table  5.2  -  Extent  of  Consensus  of 
Round  Two  for  Factors  1  and  2 


# 

Situation  Description 

Percent  of  total 
responses  in  adjacent 
categories 

Factor  1*  Factor  2** 

2 

Strengths  and  weaknesses  rated 
differently 

92.9% 

92.9% 

S 

CR/DR  rewrites  change  meaning 

71.4% 

71.4% 

6 

All  costs  unreasonable  using 
analysis  technique 

78.6% 

7  8.5% 

8 

Sticking  by  "award  without 
discussions"  policy 

78.6% 

78.6% 

12 

Not  all  deficiencies  are  discussed 
with  offerors 

78.6% 

78.6% 

14 

Bringing  new  people  into  the 
evaluation,  negotiation  phase 

85.7% 

78.6% 

^Factor  1  -  Impact  situation  has  on  the  acquisition 
program . 

**Factor  2  -  Frequency  that  situation  occurs  during  source 
selections . 


While  all  of  the  situations  tested  received  a  consensus 
response,  there  was  a  sizable  range  of  consensus  response 
rates  between  the  situations.  For  each  factor,  there  were  a 
group  of  situations  that  had  substantially  lower  consensus 
rates  and  another  group  of  that  had  substantially  higher 
consensus'  rates  than  the  other  situations  tested.  Here  are 
the  lowest  and  highest  consensus  response  rates  received  for 
each  factor: 
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Situation 

#7  - 

69.2% 

Situation 

#2 

-  92.9% 

Situation 

#3  - 

71.4% 

Situation 

#14 

-  92.9% 

Situation 

#5  - 

71.4% 

Situation 

#11 

-  92.3% 

Situation 

#9  - 

71.4% 

Situation 

#13 

-  92.3% 

Factor  2 _ ( 

Frequency) 

Situation 

#7 

-  69.2% 

Situation 

#2 

-  92.9% 

Situation 

#10 

-  69.2% 

Situation 

#15 

-  92.9% 

Situation 

#4 

-  71.4% 

Situation 

#11 

-  92.3% 

Situation 

#5 

-  71.4% 

The  situations 

with  the  lowest 

consensus  rate 

for 

either 

impact  or  frequency  may  have  less  confidence  associated  with 
concluding  they  are  pitfalls,  while  the  situations  with  high 
consensus  rates  have  a  higher  level  of  confidence. 

The  respondents  were  able  to  agree  on  both  factors  for 
all  situations  according  to  the  two-thirds  decision  rule. 
Based  on  this  it  appears  that  the  experiences  from  one 
source  selection  to  another  are  similar. 
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Table  5.3  below  shows  the  mean  response  of  each 


situation  for  both  factor  scales. 


Table  5.3  -  Mean  Responses,  Final  Results 


# 

Situation  Description 

Mean  Responses 

Vector  1*  Vector  2** 

1 

Evaluate  on  standards  not  in  RFP 

4.4 

2.4 

2 

Strengths  and  weaknesses  rated 
differently 

3.6 

2.8 

3 

Evaluators  lack  of  global  view 

3.1 

3.1 

4 

Redundant  CR/DRs 

2.6 

3.1 

5 

CR/DR  rewrites  change  meaning 

3.1 

3.0 

6 

All  costs  unreasonable  using  analysis 
technique 

4.2 

2.1 

7 

Hidden  past  performance  problems 

3.1 

2.3 

8 

4.0 

2.3 

9 

Eliminate  offers  in  competitive  range  to 
reduce  number 

3.8 

1.9 

10 

Discussing  issues  not  addressed  by  a 
CR/DR/MR 

2.5 

2.5 

11 

Technical  leveling,  transfusion, 
auctioning 

4.5 

1.8 

12 

Not  all  deficiencies  are  discussed  with 
offerors 

4.2 

2.5 

13 

BAFO  prices  exceed  funds  available 

4.4 

2.2 

14 

Bringing  new  people  into  the 
evaluation/negotiation  phase 

2.8 

4.0 

15 

Arbitrary  schedule 

3.6 

3.7 

♦Factor  1  scale  (Impact) :  1-None,  2-Minor,  3-Moderate, 
4-Significant,  5-Very  Significant 


♦♦Factor  2  scale  (Frequency) :  1-Never,  2-Seldom, 
3-Sometimes,  4-Usually,  5-Always 


These  means  were  used  in  Figure  5.1  to  show  the  relative 
position  of  each  situation  tested  using  both  the  impact 
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Figure  5.1  -  Relative  Impact/Frequency  of  Situations  Tested 


and  frequency  results  of  each  situation.  The  horizontal 
axis  measures  the  impact  the  situation  has  on  the 
acquisition  program.  The  axis  is  marked  with  the  response 
categories  for  Factor  1.  The  vertical  axis  represents  the 
frequency  with  which  these  situations  occur  in  a  source 
selection.  This  axis  is  marked  with  the  response  categories 
for  Factor  2.  To  plot  a  situation  on  this  graph,  the  mean 
for  both  impact  and  frequency  was  used.  For  example. 
Situation  #2  had  a  mean  of  3.6  for  impact  and  2.3  for 
frequency.  Its  impact  then  is  between  "moderate"  and 
"significant"  on  the  horizontal  scale,  leaning  toward 
"significant."  It  is  between  "seldom"  and  "sometimes"  on 
the  vertical  scale,  being  closer  to  "sometimes." 

Figure  5.1  shows  the  relative  impact  and  frequency  that 
these  situation  have  in  relation  to  each  other.  Movements 
on  this  graph  up  and  to  the  right  indicate  that  a  situation 
is  increasing  a  problem  to  the  acquisition  program.  For 
example.  Situation  #10  has  about  the  same  frequency  o£ 
occurrence  as  Situation  #12 .  However,  the  impact  on  the 
acquisition  program  is  much  more  severe  if  Situation  #12  is 
encountered  than  if  Situation  #10  is  encountered. 

Therefore,  acquisition  managers  should  be  much  more 
concerned  with  ensuring  all  deficiencies  are  discussed  with 
offerors  (Situation  #12)  than  with  ensuring  that  issues  not 
addressed  by  a  CF/DR/MR  are  discussed  with  offerors 
(Situation  #10) .  Conclusions  drawn  are  more  tenuous  using  a 
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similar  analysis  between  Situation  #12  and  Situation  #15. 
Although  Situation  #12  has  a  greater  impact  on  the  program. 
Situation  #15  occurs  much  more  frequently.  It  appears  that 
acquisition  managers  should  pay  attention  to  both  of  these 
pitfalls . 

Two  situations  scored  low  on  both  impact  and  frequency 
They  were  "redundant  CR/DRs"  (Situation  #4)  and  "discussing 
issues  not  addressed  by  a  CR/DR/MR"  (Situation  #10) .  They 
do  not  appear  to  happen  with  great  regularity  during  source 
selections  and  when  they  do,  their  impact  isn't  significant 
As  stated  in  Chapter  IV,  these  situation  are  not  considered 
source  selection  pitfalls.  Situation  #11,  "Technical 
transfusion,  technical  leveling  and  auctioning"  will  also 
not  be  considered  a  pitfall  for  this  study.  As  discussed 
during  the  analysis  of  this  situation  in  Chapter  IV,  it 
appears  that  there  are  multiple  issues  within  this 
situation.  As  a  result,  these  issues  should  be  separated 
and  re-tested  in  a  future  study.  The  remaining  situations 
tested  are  considered  pitfalls  in  the  source  selection 
process . 

Research  Objective  3  -  Examine  the  extent,  .to  which  these 
Pitfalls  are  common  to  particular  product  centers  or  to  all 
Air  Force  Material  Command  (AFMC)  product  centers . 

As  stated  in  Chapter  IV,  low  number  of  respondents 
from  each  of  the  four  product  centers  makes  this  research 
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objective  difficult  to  draw  conclusions.  There  did, 
however,  appear  to  be  some  differences  in  the  impact  and 
frequency  of  these  situation  between  product  centers.  As 
stated  in  Chapter  IV,  the  following  situations  may  affect 
some  product  centers  differently  than  others: 

Situation  #3.  Evaluators  lack  global  view. 

Situation  #4 .  Redundant  CR/DRs . 

Situation  #7 .  Hidden  poor  past  performance  problems . 

Situation  #8.  Sticking  by  "award  without  discussions" 
policy . 

Situation  #13.  BAFO  prices  exceed  funds  available. 

One  plausible  explanation  for  these  differences  may  be  the 
different  products  that  they  buy.  Further  research  is  first 
needed  to  validate  whether  or  not  these  differences  exist. 
Once  these  differences  are  validated,  the  reasons  for  the 
differences  can  be  further  investigated. 


Research  Obiectives  4  and  5  -  Determine  what  the 

cgn8equeflw.fe3._ar.fi ,. .to  _th.fi -acquisition  proceas-if  ans_af- th&as 

pitfalls  is  encountered  and  the  extent  to  which  Air  Force 

aaur.ce  selection  peraoflnfil  can  take  certain -steps  .Lbeat 

practices)  to  avoid  these  pitfalls  before  they  occur. 

All  of  the  possible  consequences  and  best  practices 
for  the  situations  tested  are  contained  in  Appendix  F. 

Table  5.4  summarizes  these  responses  in  template  form  for 
each  situation  that  was  determined  to  be  a  pitfall  in  this 
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study.  Each  column  is  a  separate  pitfall.  The  first  row  of 
each  column  is  the  description  of  the  pitfall.  The  second 
row  contains  the  possible  consequences  if  the  pitfall  is 
encountered.  The  third  row  is  the  frequency  with  which  the 
pitfall  occurs  and  the  last  row  contains  the  best  practices 
to  avoid  the  pitfall. 
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Table  5.4  Template  of  Consequences 


Reproduced  From 
Best  Available  Copy 


■a 

K 


« 

01 

0) 

0) 

-1.1 

-i 

& 

£ 

44 

o 

•0 

13 

M 

a 

H 

c* 

•fa 

C 

c 

tn 

01 

41 

0» 

o 

41 

O 

« 

3 

O 

•a 

a 

£ 

« 

C 

V- 

-■■X 

*>* 

u 

»r*i 

> 

41 

±> 

o 

-U 

u 

41 

M 

0 

' * 

Cl 

’•i 

:  til 

0 

4J 

d 

>1  d 

O 

o 

0i 

u 

a 

•a 

< 

3 

C5 

<44  d 

TJ 

41 

<44 

2L 

41 

r~\ 

‘;.,j 

ru 

Cn 

U 

■H  Q, 

d 

s 

O 

c* 

a 

s 

0 

U 

c 

w 

o 

41  OJ 

a 

<n 

0i 

TJ 

01 

ijl 

o 

o 

a 

• 

to 

-U 

d  0 

R 

*ri 

w 

<y\ 

V? 

*> 

u 

u 

■a 

-U 

a 

•H 

u 

01  o 

rH 

> 

4J 

O 

o 

a 

w 

® 

« 

rt 

tsi 

*6  w 

ft 

« 

44 

c 

-t 

>-W 

i 

:  *44 

41 

a 

0 

9 

Q 

•rj 

44 

•r»  -rj 

V 

41 

44 

a 

<y 

.. 

if 

•  p< 

■  1*4 

Ti 

41 

S 

Q 

TJ 

*r4 

N. 

04 

fa 

TJ 

•n 

H 

o 

d 

X 

to 

M 

U 

c 

41 

TJ 

44 

d 

>  *; 

'X 

<M 

4>-J 

u 

y 

C3 

a 

(H 

u 

<14 

•a 

O  <-4 

J3 

C 

to 

41 

. 

a 

. 

C'J 

.*4. 

a 

a 

fi 

M 

a 

O 

|U 

0) 

d 

H 

P4 

o 

V* 

K 

m. . 

C3 

<2 

a 

0 

s 

« 

n 

5 

•i-. 

>1  -rt 

0 

R 

44 

® 

a 

M 

TJ 

a 

o 

o 

fa 

:  QJ 

M 

M 

♦fl 

m 

4-** 

<44  41 

41 

3 

c 

o 

a 

CO 

Vfci 

XI 

0 

H 

•H 

p 

41 

3 

••4  C 

Q| 

41 

•H 

o 

0 

o 

-i 

o 

*2 

.H 

p 

-fa 

dj 

0 

u 

41 

13 

U  01 

a 

41 

04 

y 

44 

04 

i  j 

si 

•t: 

it 

0 

C 

5* 

fi 

> 

to 

U 

as  41 

> 

d 

<0 

44 

o 

> 

o 

-  j 

> 

«HI 

a 

si 

Cj 

a 

O 

•  a 

o 

<9 

•H 

p 

•H 

fa  O 

d 

3 

« 

44 

a 

O 

44 

yj 

/* 

Q 

L-* 

fa 

M 

TJ 

M 

X 

TJ 

ti 

at 

u  ou 

X 

a 

H 

04 

04 

X 

-U 

1. 

iU 

« 

1 

tn 

44 

<44 

Oj 

d 

0 

3 

O 

ui 

4~> 

TJ 

41 

0> 

TJ 

3  S 


d  - 

n  >1 

H 

x  <a 
n  n 
fa  o 

W  fa 
*>.  O 
14 

o 


o 

fa 


n 

C 

o 

•H 

41 

fa 

c 

fa 

<44 

■3 


<o 

44 

O 

4J 

<9 

3 

H 

<8 


d  on  c 

®  d  a 

fa 

o  tj 

41  C 
a 

O.  41 

3  n 


si 

tn 

c 

<u 

w 

41 

a 


•4 


(3 
cn 
a 

<3  TJ 
'-i  d 
H  Q) 
<9  <W 


«  a 


S  41  4J 
Odd 

jfi  «  ^ 

U  'O  4>  >  » 


»»  >»4»  »•>♦*»  •  .»«4<».»CX 


5-11 


Reproduced  From 
Best  Available  Copy 


(  ; 

c; 


o 

a 

j 

i.j 

■i 

0) 

<0 

as 

•<) 

U 


O'i 

<i. 

o 

c 

a> 

rJ 

O' 

0) 

to 

c 

o 

a 

4-t 

o 

<1) 

•P 

fij 

rH 

a 

e 

0) 

£H 


H' 


in 


o 

rH 

.Q 

ra 

Ch 


Reproduced  From 
Best  Available  Copy 


f; 

V« 

’G 

fcn 

"s 

o 

rS 

c 

■*$ 

o 

3 

•a 

.-v 

u 

*44 

14 

O 

w 

« 

w 

;i 

g 

Ci 

e 

•a 

yj 

jyi 

Si 

w 

u 

■'j 

*rf 

u 

G 

u 

X) 

r; 

Vi 

o 

o 

u 

OJ 

£J. 

c# 

■a 

3 

'3 

« 

Ci 

-•v-  r? 

JV' 

•H 

u 

o 

(0 

C 

p 

-/?•••  % 

W 

<-■{ 

•H 

3 

3 

(8 

a 

U 

1%  W' 

•  !•'- 

'  *o 

<tS 

ri 

Si 

0 

» 

O  tfj 

0 

3.* 

-a 

<3 

0) 

U  •/*. 

« 

U 

H 

TJ 

JS 

* 

tvo 

‘H 

♦  ■ 

cJ 

* 

0 

Si 

« 

5? 

tj 

n 

e 

■P 

U 

i»4 

h  a* 

to 

M 

>44 

a 

o 

4J 

u 

*"!•  •»* 

tn  o 

O 

G 

n 

0 

0 

G 

u 

a 

O 

TJ 

Q 

a 

■a 

<8 

•H 

so 

G 

P 

Q 

.a  'U 

)4 

<44 

Cl 

04 

3 

O 

<44 

o  a 

3 

V4 

q 

o 

kCI 

O 

w 

<44 

& t* 

•> 

w 

cr: 

44 

c« 

O 

P 

O 

a  a 

■rt 

w  • 
>t  3  n 
0  TJ  G 
G  _  o 

fl>  TJ  ri 
-a  O  -P 
O  83  « 
■a  O  -H 
«  H  4J 
QUO 
■o  o  ft 


m 

tfl 

«) 

a 

Cl 

<8 

•H 

■a 

TJ 

o 

G  • 

4J 

0 

c 

O 

O  TJ 

p 

* 

M 

O  * 

•a 

« 

n  u 

01  <0 

H 

:  O  Jj  o. 

u  tj 
•o 2 

§4)  Q 
o  ^ 
5  n  G 
i  a  o 

i-SS* 

>  «  p 

:s  *•* 


<U  ffl  > 

c  «  o 

o 

>«  o  « 
G  «  -a 
18  -H  TJ 

>1  G  ® 
<M  Q  H 

H  *rl  4 

p  o 

G  -H  >t 
j  <h  a 
3  4}  £ 


n 

P 

0,1 

V 

K» 

■$> 

cri 

c 

a 

TJ 

ct 

<*4 

*4 

at  <t 

[  M 

3 

■H 

P! 

» 

<44 

*44 

c 

«0 

03 

q 

a 

•a 

0 

O 

«  c 

S  5c 

•M 

■a 

G 

3 

r4  It 

<3 

H 

o  ■ 

♦k*} 

a 

G 

0 

a 

«  a: 

»  t. 

i  14 

J3 

M 

O 

<3 

p 

4J 

a 

o 

o 

>t 

M 

> 

?1  p 
t>  H  4 
■A  P  G 
P  • A  A 


H  p  Q,  Q  VI 

CJ  4t  O-rt 

4  K  H  tt 
H  Cb  <9  4) 
3  O  G  *> 

a  p  ®  o  u 

»  n  a 


4-1  U 
P  O  <0 
«  U 

<y  c  -p 

P  «S  -H 
O  JQ 

M  <44  M 

Cj  >h  a 


M  P 
O 

e  <h  4) 
o  - 
■a  *  6 

-POO 
•WHO 
W  fH 
■H  «  • 

jr  o  S 
u  «  o 

-M  l  W  C  *H 
O  « 

8-P  « 
n  3 
3*0 
WHO 

W  "t  3 


*3  O  • 
O  U 
*  0) 
.CPU 
,  -P  P  -a 

.  «  P 

>141  H 

JO  o  o 


_  w  .a 
TJ  P  a 
G  HI 
«  4)  4J 
n 

u  3  O 


n  n 

u  3  O  c 
«  B  p  o 
O  -a 

•H  G  Q  O 
P  O  Tj  « 
44  O  D  3 
O  W  P  o 
(•HD 

-rl  -W  O 
41  8  44  4> 

O  P  W 

*8  (J  «  P 

44  H  44 

M  E4  C 
G  H  M  < 

O  CQ  O  CO 

u  n  d  io 


05 

%4  l 

a 

O  J 

0 

G 

>i  O 

« 

<0 

:  Of  «> 

o 

w 

C  «' 

o 

5* 

«  to 

•H 

- 

is 

05 

-M 

O 

+1 

c 

: «-  p; 

a 

A 

o 

u  Ol 

ID 

M 

04 

o 

08  O 

m 

04 

Only  HSC  respondents  said  that  this  may  occur  and  they  were  net  ir.  complete  agreerr.er. 


Table  5.4  Template  of  Consequences  and  Best 
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Reproduced  From 
Best  Available  Copy 


Several  Conclusiona 

The  following  are  several  conclusions  that  have  been 
drawn  as  a  result  of  this  study. 

1.  There  are  many  pitfalls  that  are  common  to  source 
selections  in  the  five  AFMC  product  centers.  Survey 
respondents  reached  a  consensus  on  all  fifteen  of  the 
potential  pitfalls  tested  in  this  study.  For  each 
situation,  a  consensus  was  reached  on  its  impact  on  the 
acquisition  program  if  encountered  and  its  frequency  of 
occurrence . 

2.  Tentatively,  there  appears  to  be  a  difference  by 
product  center  on  the  impact  and  frequency  caused  by  some  of 
these  pitfalls.  The  reasons  for  these  possible  differences 
needs  further  study. 

3 .  The  situation  with  the  greatest  combined  impact  and 
frequency  was  #15,  "Trying  to  accomplish  the  source 
selection  within  an  arbitrarily  determined  schedule."  Of 
the  situations  tested,  this  is  the  one  most  likely  to  be 
encountered  that  could  have  a  significant  negative  impact  on 
the  acquisition  program. 

4.  There  is  a  possible  link  between  this  situation  and 
Situation  #8,  "Sticking  by  an  'award  without  discussions' 
policy  when  it  is  found  that  discussions  are  probably 
warranted."  The  most  frequently  cited  consequence  of 
Situation  #15  was  that  source  selection  personnel  may  rush 
the  source  selection  process  in  order  to  keep  it  on 
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schedule.  One  way  to  cut  on  the  source  selection  time 
significantly  is  to  have  no  discussions  and  award  on  the 
basis  of  the  initial  proposals  only.  Awarding  without 
discussions  may  be  one  way  that  source  selection  teams  deal 
with  arbitrary  schedules . 

Recommendations 

As  a  result  of  this  study,  the  following 
recommendations  are  made. 

1.  A  source  selection  guide,  identifying  the  pitfalls, 
consequences  and  best  practices  should  be  developed  for  use 
by  source  selection  personnel.  This  guide  could  be  similar 
to  the  Willoughby  Templates,  referred  to  in  Chapter  I. 

2 .  Current  source  selection  training  material  and 
training  should  be  amended  to  include  the  pitfalls, 
consequences  and  best  practices  identified  in  this  study. 

Limitations 

The  validity  and  applicability  of  this  research  are 
subject  to  the  following  limitations . 

1 .  The  research  participants  were  from  the  four  AFMC 
product  centers.  Therefore,  the  conclusions  drawn  are 
limited  to  the  four  product  centers .  It  is  not  known 
whether  any  of  the  findings  are  applicable  to  the  AFMC 
logistic  centers,  other  commands,  or  other  services. 
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2.  The  sample  size  for  this  research  was  small, 
consisting  of  only  14  participants.  The  background 
information,  however,  did  show  that  most  participants  had  a 
significant  amount  of  source  selection  experience. 

Therefore,  they  were  considered  appropriate  subjects  for 
this  Delphi  study . 

3.  The  research  was  limited  to  the  perspective  of 
contracting  personnel.  Their  views,  therefore,  may  be 
affected  by  parochial  interests .  The  extent  to  which  source 
selection  personrej.  from  other  disciplines  agree  with  the 
findings  of  this  study  is  not  known. 

4.  The  situations  tested  in  this  study  are  not 
intended  to  represent  an  exhaustive  list  of  the  pitfalls 
that  may  be  encountered  in  the  evaluation  and  negotiation 
phases  of  a  source  selection.  There  may  be  other  situations 
during  these  phases  that  could  have  a  significant  negative 
effect  on  an  acquisition  program. 

EfigQiamsndaiJLana  for  Eatara-Sfiafiargh 

The  following  recommendation  for  future  research  in  the 
area  of  source  selection  would  compliment  this  study. 

1.  This  study  only  looked  at  the  source  selection 
process  from  receipt  of  proposals  to  contract  award.  A 
similar  study  is  suggested  for  the  RFP  and  evaluation 
criteria  development  phases  of  a  source  selection. 
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2 .  This  study  focused  on  the  four  AFMC  product 
centers.  A  similar  study  is  suggested  for  the  AFMC 
logistics  centers.  This  replication  would  determine  the 
extent  to  which  the  pitfalls  faced  by  source  selections  in 
the  logistics  centers  are  similar  to  those  faced  by  the 
product  centers.  Similar  studies  could  also  be  accomplished 
with  personnel  from  other  disciplines  such  as  program 
management  and  engineering.  This  may  provide  a  broader 
perspective  on  the  pitfalls  encountered  in  a  source 
selection . 

3 .  This  study  could  be  replicated  with  a  larger 
sample  size.  Another  study  with  more  participants  could 
validate  the  results  of  this  research.  This  is  especially 
true  for  the  results  supporting  research  objective  three  in 
this  study.  A  large  enough  sample  from  each  of  the  four 
product  centers  could  verify  if  differences  really  do  exist 
on  the  impact  and  frequency  of  the  situations  tested  between 
the  different  product  centers. 

4.  Finally,  this  study  found  a  possible  link  between 
source  selection  schedules  and  awards  without  discussions. 

A  study  could  be  developed  to  investigate  the  extent  to 
which  there  is  a  link  between  arbitrarily  determined  source 
selection  schedules  and  the  decision  not  to  conduct 
discussions . 
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Appendix  A:  Survey  ~  Round  One 

BACKGROUND  INFORMATION 


1.  Name  _ . 

2.  Military  Rank  or  Civilian  Grade. 


3.  Office  Symbol.  _ . 

4.  AFMC  Product  Center.  _ . 

o  ASC  o  HSC 

o  ESC  o  SMC 

5.  Which  acquisition  phase  are  you  most  familiar  with? 
(may  select  more  than  one  answer) 

o  Concept  Exploration 
o  Demonstration/Validation 
o  Engineering/Manufacturing  Development 
o  Production 
o  Other  (specify) 

6.  Which  type  of  Acquisition  are  you  most  familiar  with? 
o  Aircraft 

o  Armament 
o  Electronics 
o  Space/Missile 
o  Other  (please  specify) 

7.  Years  of  Source  Selection  experience: 


o 

0-2 

years 

o 

10-15 

years 

o 

2-5 

years 

o 

15-20 

years 

o 

5-10 

years 

o 

more 

than  20  years 

8.  Number  of  Source  Selections  in  which  you  have 

participated:  _ . 

9.  Current  Air  Force  Specialty  Code  or  civilian  equivalent 

specialty:  _ . 

10.  Education  (Highest  degree  awarded), 
o  High  School  Diploma 
o  Associate  Degree 
o  Bachelors  Degree 
o  Master  Degree 
o  Doctorate  Degree 
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£afcaafcUl  Pit fills 

Potential  Evaluation  Pitfalla 


Four  the  purposes  of  this  survey  a  pitfall  is  defined  as  a 
situation  that  management  action  or  inaction  can  prevent 
that  causes  a  negative  impact  on  the  acquisition  program. 
This  impact  can  manifest  itself  as  program  delays,  reduced 
quality  of  the  resulting  contract  or  sustained  protests . 

Not  only  must  the  situation  have  a  negative  impact  on  the 
acquisition  program,  there  must  also  be  a  moderate  risk  that 
the  source  selection  will  encounter  the  situation  because  of 
the  management  action  or  inaction. 

Situation  #1:  A  technical  team  member  evaluates  a  proposal 
based  on  standards  not  in  the  Request  for  Proposal  (RFP) . 

Part.l 

Factor  1:  If  this  situation  occurs,  it's  negative  impacr  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 

Comments : 


Part  jL 

1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ... (comment) 


2 .  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment ) 
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Situation  #2 .  Similar  strengths  and  weaknesses  between 
proposals  get  rated  differently. 

Eart  1 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2 :  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Ear.t-2 

1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ...( comment ) 


2.  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to.  .(comment) 
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Situation  #3 .  Evaluators  lack  global  view  of  proposal  and 
do  not  pick  up  conflicting  or  clarifying  information. 


Fart  1 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

Nona  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Fart  2 

1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will -most  likely  be ... (comment) 


2 .  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment) 


A- 4 


Situation  #4.  High  number  of  redundant  deficiency  reports, 
clarification  requests,  and  modification  requests 
(DR/CR . /MR . )  are  generated . 

Ear£—1 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Eart  2. 

1 .  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ...( comment ) 


2 .  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment) 
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Situation  #5.  Humorous  rewrites  of  a  DR/CR/MR.  before  it  is 
approved,  sometimes  changing  its  original  meaning. 

Fart  1 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2 :  This  situation  is  { _ )  encountered  during 

a  source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


P.art  2 


1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ... (comment) 


2 .  The  be3t  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment) 
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Situation  #6.  The  price  or  costs  analysis  technique 
developed  for  the  cost/price  evaluation  renders  all  or  most 
offers  unrealistic  and  unreasonable  when  applied. 

Part  1 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


1 .  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ...( comment ) 


2.  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment) 
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Situation  #7 .  Past  performance  problems  which,  though 
relevant  to  the  source  selection,  are  unknown  to  or  hidden 
from  the  Program  Risk  Assessment  Group  (PRAG) . 


Bart  1 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2 :  This  situation  is  ( _ )  encountered  during 

a  source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Bart.  _2 

1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ...( comment ) 


2.  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment ) 
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Pot.ant.ial  Pitfalls  batwaan  Evaluation  and  Discussions 


Situation  #8.  Sticking  by  an  'award  without  discussions' 
policy  whan  it  is  found  that  discussions  are  probably 
warranted . 

Part  1 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  { _ )  encountered  during 

a  source  selection. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Part  2 

1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be. . . (comment) 


2.  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment) 
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Situation  #9.  The  motivation  during  the  competitive  range 
determination  is  to  reduce  the  number  of  offerors,  not  to 
retain  offerors  whose  proposals  have  a  reasonable  chance  of 
being  selected  for  award. 

Past  J. 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Part  2 

1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ...( comment ) 


2.  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment) 
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Potential  Discussion  Fit fall a 


Situation  #10 .  Addressing  issues  during  face  to  face 
discussions  which  were  not  addressed  with  a  DR/CR/MR. 

£act_ 1 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Eart  .2. 

1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ...  (comment ) 


2.  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment ) 
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Situation  #11.  Technical  lava ling,  transfusion  or 
auctioning  during  negotiations. 


Eart-I 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

Mona  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Part  2 

1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ... (comment) 


2.  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment) 
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Situation  #12.  Not  all  proposal  deficiencies  are  addressed 
during  discussions  with  offerors . 


Part  1 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2 :  This  situation  is  { _ )  encountered  during 

a  source  selection. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Part  2 

1 .  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ...( comment ) 


2 .  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment) 
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Potential  Jaat  *nd  final  Qffara  (BATO)  Pitfall 
Situation  #13 .  BAFO  pricos  exceed  funds  available . 


Part  1 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

Mona  Minor  Moderate  Significant  Very 

Significant 


Factor  2 :  This  situation  is  ( _ )  encountered  during 

a  source  selection. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Part  2 

1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ...( comment ) 


2 .  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment) 
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QssulaII  gotaatiai  Eitfall 


Situation  #14 .  Bringing  people  into  the 
evaluation/negotiation  phase  of  the  source  selection  who 
were  not  involved  in  the  development  of  the  RFP  and  award 
criteria . 

E,axt--1 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Eart-2 

1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ...( comment ) 


2 .  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment ) 
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Situation  #15 .  Trying  to  accomplish  the  source  selection 
within  an  arbitrarily  determined  schedule. 

Eart_l 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Comments : 


Eart  ..2 

1.  If  this  situation  is  encountered,  the  consequences  to 
the  acquisition  program  will  most  likely  be ...( comment ) 


2.  The  best  practice  a  source  selection  team  can  employ  to 
avoid  this  situation  is  to ... (comment ) 
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Appendix  B :  Round  One  Feedback 

The  following  six  situations  will  be  tested  again  in  this 
round. 

Situation  #2 .  Similar  strengths  and  weaknesses  between 
proposals  get  rated  differently. 

•Situation  #5 .  Numerous  rewrites  of  a  DR/CR  during  the 
government  review  process,  before  it  is  approved  and  issued 
to  the  offeror,  sometimes  changing  its  original  meaning. 

Situation  #6.  The  price  or  costs  analysis  technique 
developed  for  the  cost/price  evaluation  renders  all  or  most 
offers  unrealistic  and  unreasonable  when  applied. 

Situation  #8.  Sticking  by  an  'award  without  discussions' 
policy  when  it  is  found  that  discussions  are  probably 
warranted . 

Situation  #12.  Not  all  proposal  deficiencies  are  addressed 
during  discussions  with  offerors . 

Situation  #14.  Bringing  people  into  the 

evaluation/negotiation  phase  of  the  source  selection  who 
were  not  involved  in  the  development  of  the  RFP  and  award 
criteria . 

*  This  situation  was  modified  to  remove  the  acronym  MR, 
which  stands  for  Modification  Request.  Some  people  were 
unfamiliar  with  this  term.  Others  said  it  did  not  fall  into 
the  same  categories  as  CRs  and  DRs .  The  term  is  used  by  ASC 
(and  maybe  other  product  centers)  to  refer  to  an  RFP 
amendment.  An  amendment  would  not  be  issued  using  a  process 
similar  to  DR/CR  generation  and  so  was  eliminated  from  the 
situation.  Finally,  some  found  it  confusing  deciding  who 
was  doing  the  rewriting,  the  government  or  the  offerors. 

The  situation  was  also  modified  to  clarify  this. 

The  following  is  a  reminder  of  the  statements  and  scales 
used  for  Factor  1  (impact)  and  Factor  2  (frequency) . 
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Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 


1 

None 


2  3  4  5 

Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered 

during  a  source  selection  evaluation. 


1 

Never 


2 

Seldom 


3 

Sometimes 


4 

Usually 


5 

Always 


Frequency  of  Responses,  Round  1 

Situation  Group 

Number  Factor  Response  Categories  Avg. 


1 

2 

3 

4 

5 

2 

Factor 

1 

0 

2 

3 

6 

3 

3.71 

Factor 

2 

0 

5 

7 

2 

0 

2.79 

5 

Factor 

1 

2 

3 

1 

8 

0 

3.07 

Factor 

2 

1 

5 

3 

5 

0 

2.86 

6 

Factor 

1 

1 

2 

2 

3 

6 

3.79 

Factor 

2 

5 

6 

2 

1 

0 

1.93 

8 

Factor 

1 

0 

1 

4 

4 

4 

3.85 

Factor 

2 

3 

6 

4 

0 

0 

2.08 

12 

Factor 

1 

0 

1 

5 

3 

4 

3.77 

Factor 

2 

3 

3 

7 

0 

0 

2.31 

14 

Factor 

1 

0 

4 

9 

1 

0 

2.82 

Factor 

2 

0 

1 

5 

4 

4 

3.79 

Your 

Response 


The  following  comments  were  received  on  these  six  situations 
in  round  one. 

Situation  #2 :  Similar  strengths  and  weaknesses  between 
proposals  get  rated  differently. 

If  you  have  too  many  team  members  or  a  team  chief  that  is 
not  strong,  this  is  very  likely  to  occur. 
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In  a  close  race  for  the  award  this  possibly  could  make  the 
difference . 

Normally  won't  occur  if  standards  are  followed  and  [the] 
same  review  team  [is]  used. 

Similar  strengths  and  weaknesses  can  have  different  impacts 
on  offerors  depending  on  their  [each  offeror's]  individual 
approach . 

Situation  #5 :  Numerous  rewrites  of  a  DR/CR  during  the 
government  review  process,  before  it  is  approved  and  issued 
to  the  offeror,  sometimes  changing  its  original  meaning. 

A  fact  of  life  due  to  situation  #4.  (Situation  #4  was  "High 
number  of  redundant  deficiency  reports  and  clarification 
requests  (DR/CR)  are  generated."  Respondent  said  this  isn't 
always  negative  because  it  shows  that  evaluators  are  reading 
proposal.  SSET/SSEB  can  then  edit  CRs/DRs  for  consistency. 

Offerors  may  not  get  a  clear  message  as  to  what  they  should 
be  responding  to.  More  effort  [is]  expended  to  clarify  our 
questions . 

Could  lead  to  another  round  of  discussions 

Probably  nothing  [no  impact]  however,  if  the  changed  meaning 
is  the  straw  that  puts  someone  out  of  the  competitive  range 
a  protest  may  be  expected. 

Situation  #6 :  The  price  or  costs  analysis  technique 

developed  for  the  cost/price  evaluation  renders  all  or  most 
offers  unrealistic  and  unreasonable  when  applied. 

[May]  require  an  amendment  to  the  RFP  and  new  cost 
proposals . 

No  award  possible  without  significant  discussions.  RFP 
amendment  or  cancellation  may  be  necessary. 

Situation  #8:  Sticking  by  an  'award  without  discussions' 
policy  when  it  is  found  that  discussions  are  probably 
warranted . 

Discussions  have  been  held  in  the  majority  of  cases  I  have 
been  involved  in . 

Can  be  a  big  problem  when  evaluators  are  intimidated  by 
pressure  from  the  top  to  award. 
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[If  contract  is  awarded  under  these  conditions]  then  [the 
Air  Force  is]  forced  to  rectify  shortcomings  in  a  sole 
source  environment . 

Situation  #12:  Not  all  proposal  deficiencies  are  addressed 

during  discussions  with  offerors . 

Need  good  review  to  assure  this  doesn't  happen. 

A  deficiency  makes  an  offeror  unawardable. 

Probably  results  in  [a]  poor  competitive  range  determination 
prior  to  BAFO. 

Situation  #14 :  Bringing  people  into  the 
evaluation/negotiation  phase  of  the  source  selection  who 
were  not  involved  in  the  development  of  the  RFP  and  award 
criteria . 

Evaluators  do  not  know  what  to  look  for  and  what  [the]  award 
criteria  is.  A  lot  of  effort  [is]  expended  during  source 
selection  to  bring  [these]  people  up  to  speed. 

My  experience  has  been  that  you  are  dealing  with  a  small 
number  of  people  in  the  development  phase  and  a  larger  group 
in  the  evaluation  phase. 

Insignificant  if  they  are  well  trained. 

No  team  buy  in.  Problems  with  evaluation  due  to  lack  of 
understanding . 

Normally  on  larger  source  selections,  additional  parties  are 
brought  into  the  process  for  "greybeard"  input.  Normally 
these  are  not  big  issues  and  don't  impact  the  process. 

Usually  a  way  of  life  as  only  [a]  limited  number  of  people 
are  involved  in  the  development  of  the  RFP . 

May  slow  down  the  source  selection  process  but  [I]  do  not 
believe  it  is  a  significant  problem. 
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Appendix  C :  Survey  -  Round  Two 


Situation  #2 .  Similar  strengths  and  weaknesses  between 
proposals  get  rated  differently. 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Situation  #5.  Numerous  rewrites  of  a  DR/CR  during  the 
government  review  process,  before  it  is  approved  and  issued 
to  the  offeror,  sometimes  changing  its  original  meaning. 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 

Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  #6.  The  price  or  costa  analysis  technique 
developed  for  the  cost/price  evaluation  renders  all  or  most 
offers  unrealistic  and  unreasonable  when  applied. 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2 :  This  situation  is  ( _ )  encountered  during 

a  source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Situation  #8.  Sticking  by  an  ’award  without  discussions' 
policy  when  it  is  found  that  discussions  are  probably 
warranted. 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2 :  This  situation  is  ( _ )  encountered  during 

a  source  selection. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  #12.  Not  all  proposal  deficiencies  are  addressed 
during  discussions  with  offerors . 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Situation  #14 .  Bringing  people  into  the 
evaluation/negotiation  phase  of  the  source  selection  who 
were  not  involved  in  the  development  of  the  RFP  and  award 
criteria . 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


Factor  2:  This  situation  is  ( _ )  encountered  during 

a  source  selection. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Frequency 


Appendix  D :  Final  Results  for  Factors  1  (Impact) 

and  2  l Frequency  ) 


Situation  »1  r  A  technical  team  member  evaluates„a^_proPQsal 
baaed  on  standards  not  in  the  Request  fox  Propo3al_(RFP) . 

Situation  1 ,  Factor  1 


significant 


■  SMC 

■  HSC 

□  esc 

□  ASC 


Response  Category 


Figure  D.l 


Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 


1 

None 


2 

Minor 


3 

Moderate 


4  5 

Significant  Very 

Significant 
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Frequency 


Situation  1,  Factor  2 


never  seldom  sometimes  usually  always 

Response  Category 


Figure  D.2 


Factor  2:  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Frequency 


Situation  2,  Factor  1 


7  - 

6 - 

5 - 

4 - 

3  - 

2  - 

1 - 

0 - 

none 


■  SMC 

a  hsc 

□  ESC 

□  ASC 


minor  moderate  significant  very 

significant 


Response  Category 


Figure  D.3 


Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 


1 

None 


2 

Minor 


3 

Moderate 


4 

Significant 


5 

Very 

Significant 
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Frequency 


Situation  2,  Factor  2 


never  seldom  sometimes  usually  always 
Response  Category 

Figure  D.4 


Factor  2:  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


situation  Evaluators  iacn  giopaj.  view  or  proposal  ana 

do  not  pick  up  conflicting  or  clarifying  information. 


Situation  3,  Factor  1 


significant 


■  SMC 

■  HSC 

□  ESC 

□  ASC 


Response  Category 


Figure  D.5 


Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 


1 

None 


2  3  4  5 

Minor  Moderate  Significant  Very 

Significant 
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Frequency 


Situation  3,  Factor 


never  seldom  sometimes  usually  always 
Response  Category 


Figure  D.6 


Factor  2:  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  4,  Factor  1 


none  minor  moderate  significant  very 

significant 

Response  Category 


Figure  D.7 


Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 
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Situation  4,  Factor  2 


never  seldom  sometimes  usually  always 
Response  Category 

Figure  D.8 


Factor  2 :  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  5,  Factor  1 


none  minor  moderate  significant  very 

significant 

Response  Category 


Figure  D.9 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 
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Situation  5,  Factor  2 


Response  Category 


Figure  D.10 


Factor  2:  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  6,  Factor  1 


□  ESC 

□  ASC 


minor  moderate  significant  very 

significant 

Response  Category 

Figure  D.ll 


Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 


1 

None 


2 

Minor 


3  4  5 

Moderate  Significant  Very 

Significant 


D-ll 


Frequency 


Situation  6,  Factor  2 


never  seldom  sometimes  usually  always 

Response  Category 


■  SMC 

■  HSC 

□  ESC 

□  ASC 


Figure  D.12 


Factor  2:  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  7,  Factor  1 


none  minor  moderate  significant  very 

significant 

Response  Category 


Figure  D.13 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 
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Situation  7,  Factor  2 


never  seldom  sometimes  usually  always 


■  SMC 

■  HSC 
D  ESC 
□  ASC 


Response  Category 


Figure  D.14 


Factor  2 :  This  situation  { _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Frequency 


Situation  8,  Factor  1 


■  SMC 

■  HSC 

□  esc 

□  ASC 


none  minor  moderate  significant  vary 

significant 


Response  Category 


Figure  D.15 


Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 
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Frequency 


Situation  8,  Factor  2 


never  seldom  sometimes  usually  always 


Response  Category 

Figure  D.16 

Factor  2 :  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation . 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  <9.  The  motivation  during  the  competitive  range 
determination  is  to  reduce  the  number  of  offerors,  not  to 
retain  offerors  whose  proposals  have  a  reasonable  chance  of 
being  selected  for  award. 


Situation  9,  Factor  1 


significant 


■  SMC 

■  HSC 

□  ESC 

□  ASC 


Response  Category 


Figure  D.17 


Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 


t 
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Frequency 


Situation  9,  Factor  2 


6 


never  seldom  sometimes  usually  always 

Response  Category 

Figure  D.18 


Factor  2:  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  1 0,  Factor  1 


none  minor  moderate  significant  very 

significant 

Response  Category 

Figure  D.19 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 
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Frequency 


Situation  10,  Factor  2 


5 

4.5 

4 

3.5 

5 

2.5 
2 

1.5 
1 

0.5 
0 

never  seldom  sometimes  usually  always 

Response  Category 

Figure  D.20 


Factor  2 :  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 


Situation  1 1 ,  Factor  1 


none  minor  moderate  significant  very 

significant 

Response  Category 


Figure  D.21 

Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 
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Frequency 


Situation  1 1 ,  Factor  2 


never  seldom  sometimes  usually  always 
Response  Category 


Figure  D.22 


Factor  2:  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4-5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  1 2,  Factor  1 


none  minor  moderate  significant  very 

significant 

Response  Category 

Figure  D.23 

.  Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  3  4  5 

None  Minor  Moderate  Significant  Very 

Significant 
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Frequency 


Situation  12,  Factor  2 


never  seldom  sometimes  usually  always 
Response  Category 

Figure  D.24 


Factor  2:  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  1 3,  Factor  1 


none  minor  moderate  significant  very 

significant 

Response  Category 

Figure  D.25 


Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 

1  2  '  3  4  5 

None  Minor  -  Moderate  Significant  Very 

Significant 
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Situation  1 3,  Factor  2 


never  seldom  sometimes  usually  always 

Response  Category 


■  SMC 

■  HSC 

□  ESC 

□  ASC 


Figure  D.26 


Factor  2 :  This  situation  { _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  1 4,  Factor  1 


none  minor  moderate  significant  very 

significant 


■  SMC 

■  HSC 

□  ESC 

□  ASC 


Response  Category 


Figure  D.27 


Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 


1 

None 


2 

Minor 


3 

Moderate 


4 

Significant 


5 

Very 

Significant 


D-27 


Frequency 


Situation  14,  Factor  2 


never  seldom  sometimes  usually  always 
Response  Category 


Figure  D.28 


Factor  2 :  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Situation  #15, _ Trying  to  accomplish  the  source  selection 

within  an  arbitrarily  determined  schedule. 

Situation  1 5,  Factor  1 


none  minor  moderate  significant  very 

significant 

Response  Category 


Figure  D.29 


Factor  1:  If  this  situation  occurs,  it's  negative  impact  on 
the  acquisition  program  is  likely  to  be  ( _ )  . 


1 

None 


2 

Minor 


3 

Moderate 


4 

Significant 


5 

Very 

Significant 
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Situation  1 5,  Factor  2 


never  seldom  sometimes  usually  always 
Response  Category 


H  SMC 

■  HSC 

□  ESC 

□  ASC 


Figure  D.30 


Factor  2:  This  situation  ( _ )  encountered  during  a 

source  selection  evaluation. 

1  2  3  4  5 

Never  Seldom  Sometimes  Usually  Always 
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Appendix  E : 


Background  Informatic 


Table  E.l 

Military  Rank  or  Civ 

Rank  or  Grade 

COL 

CAPT 

GM-15 

GM-14 

GM-13 

GS-12 

TOTAL 


Table  E . 2 

fligfasat. . SducatlQj 

Degree 

Bachelors 

Masters 

Doctorate 

TOTAL 


Table  E.3 


Current  Product  Cent 
Respondents  are  Wc 


Product-Center 

ASC 

ESC 

HSC 

SMC 


TOTAL 


Table  E . 4 


Current  AFSC  Specialty  Code 
or  Civilian  Equivalent 


AFSC 

1102 

1101 

64A4 

xxxx 

TOTAL 


Mumhar  of  Respondents 

10 

1 

2 

1 

14 


Table  E.5 

Years  of  Source  Selection  Experience 


Experience  (Years) 

0-2 

2-5 

5-10 

10-15 

TOTAL 


Number,  of  Respondents 

2 

3 

1 

2  • 

13???? 


Table  E.6 

Number  of  Source  Selections  (SS) 

Respondent  has  Participated  In 

Number  of  SS  Number  of  Respondents 

1-5 
6-15 
16-30 
31-50 
51+ 

TOTAL 


7 

2 

2 

1 

2 

14 
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Table  E.7 


Phase 

Concept  Exploration 
Demonstration/Validation 
Engineering  Manufacturing 
Development 
Production 
Other 

**  NOTE:  MULTIPLE  RESPONSES  WERE  POSSIBLE 


Table  E . 8 


Type 

Aircraft 
Armament 
Electronics 
Space/Miisile 
Other 

**  NOTE:  MULTIPLE  RESPONSES  WERE  POSSIBLE 
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Appendix  F :  Responses  to  the  Consequences  and  Best 
Practices  for  Each  Situation  Tested 


Situati  #1: _ A  technical  team  member  evaluates  a  proposal 

based  on  standards  not  in  the  Request  for  Proposal  (RFP )  . 

Consequences : 

Not  awarding  to  the  most  qualified  contractor,  protest 
potential  we're  sure  to  lose;  time  consuming  because  re- 
evaluation  necessary. 

Sustained  protest.  All  offerors  should  be  compared  against 
pre-defined  standards. 

Have  to  perform  evaluation  again  IAW  standards . 

Corrected  during  the  process  by  SSET  review. 

Best  Practices : 

Ensure  evaluators  have  a  copy  of  the  standards,  understand 
the  evaluation  process;  buyer/CO  and  SSET  chairman  monitor 
evaluations  early  to  correct  this  situation  ASAP  and  get  the 
evaluators  back  on  track. 

Train  team  to  evaluate  against  standards .  Ensure 
chairperson  reviews  all  weaknesses  and  strengths  against 
standards . 

Up  front  briefing.  Oversight  by  team  chief.  Written 
documentation . 

Have  briefings  by  the  CO/SSET  chairman  on  procedures.  Make 
sure  team  chiefs  have  good  understanding  of  the  process . 

Have  the  team  chief,  chairman  and  CO  thoroughly  review 
evaluations . 

Have  each  technical  member  evaluate  only  one  element . 

Select  technical  team  membership  using  higher  criteria  than 
just  having  experience  or  involvement  in  the  program.  Need 
some  sort  of  prerequisite  training. 

Review  evaluation  write-ups  and  challenge  those  that  are 
not,  or  do  not  appear  to  be  based  on  the  evaluation 
standards . 
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Situation  #2  . _ Similar  strengths  and  weaknesses  between 

proposals  get  rated  differently. 


Consequences : 

Not  significant  unless  overall  ratings  are  very  close.  This 
should  alert  management  to  do  a  quality  review  on  these 
proposals . 

This  makes  selection  for  SSA  difficult  and  confusing;  time 
consuming  because  re-evaluation  required. 

Sustained  protest,  if  awardee  had  same  weakness  but  was 
rated  differently. 

Moderate,  if  the  strengths  are  in  items /factors  that  are 
more  important. 

Award  could  be  slanted  towards  incumbent  (if  performance  and 
relations  with  the  technical  members  are  good) . 

Don't  have  more  than  one  item  captain  for  each  item,  even  if 
you  have  a  large  quantity  of  proposals . 

Best  Practices: 

Have  the  same  evaluator  evaluate  all  proposals  for  one 
factor,  then  review  the  strengths  and  weaknesses  for  all  and 
associated  ratings  for  consistency.  CO/buyer  and  SSET 
Chairman  as  well  as  area/factor  captains  review  area/factor 
and  summaries  for  consistency. 

Have  someone  review  across  proposals  to  ensure  consistency 
of  evaluations.  Follow  color/risk  rating  definitions 
closely . 

Evaluators  should  be  challenged  to  defend  write-up  to  ensure 
they  have  a  clear  understanding  of  what  is  a  weakness  or 
what  is  a  strength. 

Use  members  that  are  not  closely  involved  with  incumbent. 
This  may  not  be  possible.  Standards  could  be  stated  as  such 
that  they  are  more  objective. 


Situation  #3  . _ Evaluators  lack  global  view  of  proposal  and 

do  not  pick . up  .conflicting  or  clarifying  information. 

Consequences : 
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Award  goes  to  contractor  not  qualified  or  visa  versa.  May 
result  in  problems  after  award. 

The  best  offeror  may  not  be  selected. 

Contract  will  have  an  increase  in  modifications  after  award. 

Not  significant  as  areas  are  rolled  up  to  form  any  overall 
assessment  these  differences  will  surface. 

If  conflicting  information  is  overlooked,  could  award 
contract  and  discover  you  have  disputed  contractual 
requirements . 

Best  Practices: 

Have  SSET  meetings  to  discuss  CR/DRs .  Attendees  must  be 
familiar  enough  with  their  factors  to  clarify  or  identify 
potential  discrepancies.  I  don't  think  it  is  realistic  to 
have  each  evaluator  read  all  areas  in  a  proposal . 

Train  the  team  to  key  in  on  important  factors  to  select  the 
best  offeror.  Ensure  chairperson  reviews  and  questions  what 
team  discovers. 

Get  technical  and  management  teams  to  at  least  read/review 
other  pertinent  parts  of  proposal  and  have  good  team  cross¬ 
talks.  Everyone  must  function  as  a  team. 

Have  one  member  of  the  SSET  review  a  proposal  holistically 
while  others  review  by  assigned  elements. 

Use  a  few  experienced  personnel  who  can  read  and  digest 
entire  proposal.  Call  for  page  limits  on  proposals.  Hold 
daily  team  or  area  meetings  to  discuss 
findings/issues/CR/DRs . 

Have  the  evaluators  read  the  entire  proposal  and  not  just 
their  little  section. 


Situation  #4.  High  number  of  redundant- deficiency  reports 
and  clarification  requests  (DR/CRs)  _ .are -generated  . 


Consequences : 

Mostly  a  delay  in  award  because  of  the  need  for  extensive 
discussions  to  clean  up  the  CR/DRs. 


Not  significant.  Offerors  will  answer  same  questions  twice. 

This  is  not  always  an  indication  of  a  negative  result.  It 
shows  that  the  evaluators  are  reading  proposal .  SSET/SSEB 
can  then  edit  CR/DRs  for  consistency. 

Easy  to  fix.  I  would  rather  have  redundant  CR/DRs  than  fail 
to  identify  problems  to  an  offeror. 

Delay  in  reaching  a  decision  and  increased  risk  of  "re¬ 
proposals"  via  the  CR/DR  process. 

Delays  the  acquisition  and  does  not  add  value  to  the 
process . 

Minor;  CO  and  SSA  must  filter  out. 

Serves  as  a  cross  check  that  the  team  is  appropriately 
evaluating  against  the  standards . 

Lots  of  extra  time  and  impact  to  schedule  for  award. 

Best  Practices : 

Issue  a  draft  RFP  and  have  a  pre-proposal  conference.  The 
reason  for  redundant  CR/DRs  is  because  something  in  the  RFP 
was  not  clear.  Provide  a  vehicle  for  contractors  to  review 
the  RFP  and  ask  questions  up  front.  This  should  reduce  the 
number  of  CR/DRs . 

Have  chairperson  review  for  duplication.  Consolidate  like 
items .  Have  team  work  together  to  create  CR/DRs . 

Aggressive  reviews  by  the  PCO  and  SSET  chairperson  prior  to 
release  of  any  CR/DR. 
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Situation  #5.  Numerous  rewrites  of  a  DR/CR  during  the 
government  review  process,  before  it  is  approved  and  issued 
to  the  offeror,  sometimes  changing  its  original  meaning. 

Consequences : 

Failure  to  conduct  meaningful  discussions . 

May  be  a  delay  in  issuing  CR/DRs,  potential  delay  in  award. 
If  the  meaning  is  changed,  it  is  obvious  the  true  concern 
will  not  be  addressed.  Then  the  question  becomes,  do  we 
issue  more  CR/DRs  or  end  discussions  and  the  offeror  suffers 
as  a  result  of  our  error. 

The  offerors  may  not  get  a  clear  message  as  to  what  they 
should  be  responding  to.  More  effort  expended  to  clarify 
our  response. 

Consequence  is  positive.  It  shows  SSET/SSEB  is  doing  their 
job  and  makes  for  a  better  product  when  CR/DRs  are  issued. 

The  real  deficiency  or  clarification  is  not  answered. 

The  original  deficiency  becomes  obscured. 

Not  getting  the  response  you  expected  and  having  to  enter 
another  round  of  discussions. 

Probably  nothing.  However,  if  this  changed  meaning  is  the 
straw  that  puts  someone  out  of  the  competitive  range  a 
protest  may  be  expected. 

Best  Practices: 

Train  evaluators  on  how  to  write  CR/DRs.  If  a  rewrite  is 
necessary,  the  evaluator  should  be  involved  during  the 
rewrite  process  to  ensure  the  meaning  isn't  changed. 

Team  effort  to  discuss/write-up  CR/DRs.  Chairperson  must 
establish  clear  guidance.  Evaluate  all  write-ups  in  draft 
form  before  finalizing. 

I  don't  think  you  want  to  avoid  this  situation. 

Have  technical  expert  review  it  before  it  goes  out . 

Avoid  too  many  review  levels .  Ensure  that  the  CR/DR  agrees 
with  what  was  in  the  RFP  and  the  evaluation. 
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offers  unrealistic  and  unreasonable  when  applied. 

Consequences : 

May  not  be  able  to  justify  award  of  best  offeror.  SSA 
cannot  make  a  decision  based  on  the  analysis . 

Re-solicitation  and  significant  delays.  Restructuring  of 
program . 

This  would  require  an  amendment  to  the  RFP  and  new  cost 
proposals . 

If  it  is  not  a  government  estimate  problem,  the  RFP  must  be 
unclear. 

An  uninformed  decision  and  poor  contract  cost  performance. 

No  award  possible  without  significant  discussions .  RFP 
amendment  or  cancellation  may  be  necessary. 

Best  Practices: 

Establish  reasonable  assessments  of  costs  associated  with 
proposals .  Review  against  funding  available  to  ensure 
offers  are  affordable. 

Use  draft  RFP  and  pre-solicitation  conferences . 

Work  on  government  estimate  early  to  get  out  all  the  kinks . 
Use  only  validated  price/cost  analysis  techniques. 


Situation  #7 . _ East  performance  .problems  which,  though 

relevant  to  the  source  selection,  are  unknown  to  or  hidden 
from  the  Program  Risk  Assessment  Group  (PRAG) . 

Consequences : 

Offeror  may  be  selected  without  proper  risk  identified  on 

past  performance.  * 
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Best  Practices: 


Ensure  source  selection  team  identifies  all  past  performance 
issues  to  PRAG.  Ensure  PRAG  gets  proper  information  from 
on-going  acquisition  programs . 

Get  information  from  as  many  sources  as  possible. 

Delve  deep.  Draft  acquisition  unique  questionnaires.  Look 
beyond  CPARS  and  what  the  offeror  tells  you. 


Situation  #8.  Sticking  by  an award  without  discussions  * 
policy  when  it  is  found  that  discussions  are  probably 
warranted . 

Consequences : 

A  contract  that  is  not  as  good  as  it  can  be. 

Are  discussions  warranted  because  every  proposal  has  a  CR/DR 
or  because  it  is  in  the  best  interest  of  the  government .  In 
the  latter  case,  it  could  result  in  selection  of  a  lesser 
qualified  company.  Sometimes  the  government  is  better  off 
not  opening  discussions,  especially  in  a  service  contract 
source  selection. 

We  do  not  get  the  system  we  believed  we  were  getting. 

Probably  not  encountered  until  after  award  when  SPO  realizes 
what  they  failed  to  understand  as  part  of  offeror's 
proposal . 

Possible  award  to  wrong  offeror. 

An  increased  likelihood  of  post  award  modifications  due  to 
ECPs  and  CCPs . 

Offer  may  not  support  the  best  needs  of  the  government. 

Technical,  schedule  and  cost  problems  once  on  contract. 
Expensive  fixes  because  you've  lost  the  competitive 
environment . 


F-7 


Best  Practices: 


Do  not  award  without  discussions  if  all  items  are  not 
clearly  identified  and  a  clear  understanding  is  not 
obtained . 

Avoid  unreasonable  time  constraints . 

Ensure  that  sufficient  acquisition  milestones  are  in  place 
to  allow  for  discussions  and  that  any  needed  discussions  are 
held.  Don't  rush  the  process  if  it  is  detrimental  to  the 
final  award. 

PCO  must  be  adamant  in  requesting  permission  from  the  SSA  to 
enter  into  discussions. 

Source  selection  team  chief  must  have  morale  fortitude  to 
tell  SSA  that  discussions  are  absolutely  essential  and  back 
up  assertion  with  rationale. 

Make  sure  that  the  evaluators  are  instructed  to  write  up  all 
CR/DRs . 

If  in  doubt,  spend  time  up  front  to  avoid  problems  later. 

Use  the  competitive  environment  to  negotiate  changes . 


Situation  *9.  The  motivation  during  the  competitive  range 
determination  is  to  reduce  the  number  of  offerors,  not  to 
retain  offerors  whose  proposals  have  a  reasonable  chance  of 
being  selected  for  .ajgajaL. 

Consequences : 

Protest  may  be  sustained  if  an  offeror  is  arbitrarily 
eliminated. 

Best  Practices: 

Ensure  only  those  offerors  who  don't  have  a  reasonable 
chance  for  award  are  removed  from  the  competitive  range. 
Ensure  evaluations  are  accurate  and  provide  offerors  with 
all  CR/DRs  with  letter  that  tells  them  they  are  not  in  the 
competitive  range. 

Have  all  PCO  determinations  to  exclude  offers  from  the 
competitive  range  reviewed  by  the  JAG  office  and  the  staff 
source  selection  officer. 
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Situation  #10 .  Addressing  issues  during  face  to  face 
discussions  which  were  not  addressed  with  a  DR/CR/MR. 


Consequences : 

Technical  people  have  a  tendency  to  bring  up  issues  not  part 
of  the  CR/DR. 

Offeror  may  protest  that  they  were  unfairly  evaluated  and 
did  not  have  time  to  address  issues  presented. 

Additional  time  for  offeror  to  document  additional 
discussion  issues.  This  puts  initial  evaluation  at  risk. 

But  better  to  catch  it  now  than  to  never  discuss  it. 

Possible  technical  leveling. 

Most  important  caveat  is  meaningful  discussions.  In  other 
words,  better  late  than  never. 

Best  Practices: 

Train  people  on  etiquette  for  discussions  (face-to-face) . 

Allow  offeror  time  to  address  any  new  discoveries  at  face- 
to-face  follow-up  with  a  CR/DR.  Ensure  team  completes 
thorough  evaluation  prior  to  calling  face-to-face 
discussions. 

Team  chiefs  need  to  do  traceability  of  write-ups  .to  CR/DRs . 

Answer  all  question  in  writing  to  all  offerors . 

Write  down  a  discussion  guide  prior  to  the  face  to  face 
meeting,  cross  reference  it  to  the  CR/DR  and  stick  to  the 
guide  during  the  discussion. 


Situation  #11.  Technical  leveling,  transfusion  or 
auctioning  during  negotiations. 

Consequences : 

Unsuccessful  offerors  may  protest  award. 


Best  Practices: 
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Evaluate  against  standards  only,  not  against  other 
proposals.  Ensure  discussions  at  face-to-face  only  address 
things  in  the  offeror's  proposal.  Train  team  members  to 
ensure  they  understand  rules/role  in  face-to-face 
discussions . 

Control  who  sits  in  on  discussions.  Carefully  screen 
CR/DRs .  Ensure  no  side  discussions  take  place. 

Ensure  CR/DRs  adequately  address  the  weakness/deficiency 
without  leveling  or  transfusion.  CO  and  committee  review 
and  coordination  can  prevent  this . 


Situation  #12.  Not  all  proposal  deficiencies  are  addressed 
during  discussions  with  offerors^ 

Consequences : 

The  problem  occurs  when  offerors  are  treated  differently, 
not  addressing  all  DRs  with  one  contractor  but  doing  so  with 
another  contractor. 

Failure  to  conduct  meaningful  discussions. 

Sustained  protest. 

A  misunderstanding  of  proposal  versus  government 
requirement . 

An  increased  likelihood  of  ECPs  after  award. 

Canceled  solicitation  or  second  round  of  discussions  and 
RAFOs. 

A  protest  from  an  unsuccessful  offeror  when  he  sees  at  the 
debriefing  a  deficiency  that  was  not  disclosed  to  him  during 
negotiations . 

Best  Practices : 

Review  and  compare  weaknesses/deficiencies  with  CRs  and  DRs 
prepared . 

Ensure  all  things  were  addressed  before  completing 
discussions.  Identify  any  new  deficiencies  as  soon  as  they 
are  discovered. 
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Easy  to  avoid  if  DRs  were  initially  documented  if  a  log  is 
kept.  Establish  an  agenda  for  discussions  and  stick  to  it. 
Have  contractor  submit  minutes  after  each  session. 

Have  SSET  leader  do  a  careful  quality  review. 

Limit  areas  to  be  evaluated  to  those  things  that  are 
critical  to  performance. 


Si.tua.tiQn  .113  . _ BAFO  prices  exceed  funds  available. 

Consequences : 

Unable  to  award  contract . 

Second  round  of  discussions  and  BAFOs  or  additional  funds 
obtained . 


Best  Practices : 

If  funds  are  significantly  constrained,  provide  funding 
profile  to  all  offerors  with  RFP  to  ensure  an  award  can  be 
made. 

Consider  conducting  a  "Best  Value"  procurement. 

Prepare  the  government  estimate  using  the  same  methods  to  be 
used  in  the  source  selection  evaluation. 

In  some  circumstances  share  funding  profile  on  years  that 
are  constrained.  Set  forth  funding  constraints  in  Section 
L.  Discuss  this  during  discussion  and  reiterate  in  BAFO 
request . 

The  government  should  disclose  available  funds  for  any  final 
year  that  is  critical  and  state  in  the  proposal  preparation 
instructions  that  the  offeror  must  comply  with  any  funding 
constraints  noted  in  the  RFP . 


Situat  j.Qa-&I4  , _ Bringing  people  into  the 

evaluation/negotiation  phase  of  the  source  selection  who 
were  not  involved  in  the  development  of  the  RFP  and  award 
criteria. 

Consequences : 
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A  bigger  learning  curve  and  possibly  a  longer  acquisition 
time  frame. 

Evaluators  not  understanding  or  agreeing  with  RFP 
requirements  or  award  criteria. 

Evaluators  do  not  know  what  to  look  for  and  what  the  award 
criterion  is .  A  lot  of  effort  is  expended  during  the  source 
selection  to  bring  people  up  to  speed. 

Minor  if  individual  has  been  briefed  on  program  and 
understands  requirements  and  has  read  proposals  and 
solicitation. 

No  team  buy  in.  Problems  with  evaluation  due  to  lack  of 
understanding . 

Added  time  to  get  new  people  up  to  speed.  Usually  a  way  of 
life  as  only  a  limited  number  of  people  are  involved  in  the 
development  of  the  RFP . 

Best  Practices: 

We  brought  people  in  because  of  sheer  volume  of  proposals 
received.  It  took  us  a  month  to  get  the  technical  team  on 
track.  Additionally,  most  had  no  source  selection 
experience . 

In  a  perfect  world,  the  people  who  developed  the  RFP  would 
also  evaluate  the  proposals  submitted  in  response  to  it . 

Keep  team  together  from  RFP  through  award.  New  members  must 
be  trained  before  evaluation  of  proposals .  Ensure  key 
players  understand  requirements  and  are  responsible  for 
screening  all  evaluations . 

Evaluate  stability  when  selecting  team  members . 

Normally  on  larger  source  selections  additional  parties  are 
brought  into  the  process  for  greybeard  input.  Normally 
these  are  not  big  issues  and  don't  impact  the  process. 

Situation  #15. _ Trying  to  accomplish  the  source  selection 

within  an  arbitrarily  determined  schedule. 

Consequences : 

Work  may  not  be  complete  on  time,  schedule  slips. 

Unhappy  customers  and  offerors . 
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Source  selection  periods  are  normally  somewhat  arbitrarily 
set  based  on  user  needs  and  resource  allocations . 

May  affect  morale  of  employees . 

May  have  inadequate  time  to  effectively  evaluate  all 
proposals.  May  award  contract  to  a  lesser  qualified 
offeror . 

Best  Practices : 

Use  the  IASP  procedures  and  closely  monitor  schedule. 

Base  schedule  on  realistic  mile-stones.  Establish  daily 
tasks  for  team  members  to  complete.  If  schedule  is 
unachievable,  reevaluate  and  set  up  realistic  schedule. 

Ensure  realistic  milestones  and  allow  for  some 
contingencies . 

Plan  for  delays  throughout  the  program. 

Don't  be  too  optimistic. 

Make  sure  the  proposal  schedule  has  been  reviewed  by  the 
center  source  selection  offeror. 
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13.  ABSTRACT  (Maximum  200  words) 


Source  Selection  is  the  process  used  by  the  Federal  government  to  select  contractors  for  large,  complex  and  important 
requirements.  Recent  studies  have  indicated  that  the  inexperience  of  source  selection  personnel  continues  to  be  a 
significant  problem  in  source  selections.  Air  Force  Federal  Acquisition  Regulation  Appendix  AA  requires  that  lessons 
learned  be  compiled  at  the  conclusion  of  every  source  selection.  Unfortunately,  there  has  been  little  success  in  organizing 
this  information  into  a  coherent  training  guide.  This  research  attempts  to  fill  this  gap  by  using  source  selection  experts  to 
identify  source  selection  pitfalls,  their  consequences,  and  the  best  practices  to  avoid  them.  Fifteen  situations  that  could  be 
detrimental  to  a  source  selection  were  identified  through  a  literature  review.  Next,  source  selection  experts  were  surveyed 
s  to  see  if  they  could  agree  on  the  nature  and  affect  these  situations  had  on  source  selections.  The  experts  evaluated  each 

situation  in  terms  of  its  negative  impact  on  an  acquisition  program  and  its  frequency  of  occurrence.  The  experts  were 
:  surveyed  using  the  Delphi  method,  characterized  by  iterative  survey  rounds  and  feedback  from  previous  rounds.  The  two 
*  factors  of  impact  and  frequency,  as  well  as  expert  comments,  were  used  to  identify  which  situations  were  pitfalls.  Twelve 

of  the  fifteen  situations  tested  were  found  to  be  source  selection  pitfalls.  The  experts  also  provided  the  possible 
consequences  of  and  best  practices  to  avoid  each  pitfall. 
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